[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894810715282-4045-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:46:50 dist-test-slave-m9wg krb5kdc[4052](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:46:50 dist-test-slave-m9wg krb5kdc[4052](info): set up 2 sockets
Jun 19 18:46:50 dist-test-slave-m9wg krb5kdc[4052](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260619 18:46:51.765200 4045 mini_kdc.cc:121] Time spent starting KDC: real 0.968s user 0.000s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894810715282-4045-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894810715282-4045-0/krb5kdc/test-user.keytab.
Jun 19 18:46:51 dist-test-slave-m9wg krb5kdc[4052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894811, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:46:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:46:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.243.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab.
Entry for principal kudu/127.3.243.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab.
I20260619 18:46:51.920138 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894810715282-4045-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894810715282-4045-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894810715282-4045-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.126:34147
--webserver_interface=127.3.243.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab
--principal=kudu/127.3.243.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:38057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.126:34147
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:46:52.230509 4065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:46:52.230773 4065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:46:52.230903 4065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:46:52.238090 4065 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:46:52.238193 4065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:46:52.238265 4065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:46:52.238341 4065 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:46:52.238406 4065 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:46:52.247944 4065 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:38057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894810715282-4045-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894810715282-4045-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.243.126:34147
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab
--principal=kudu/127.3.243.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.126:34147
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.243.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894810715282-4045-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:46:52.249236 4065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:46:52.250559 4065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:46:52.261984 4070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:52.262042 4071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:52.262043 4073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:46:52.262567 4065 server_base.cc:1034] running on GCE node
I20260619 18:46:52.263243 4065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:46:52.264583 4065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:46:52.265793 4065 hybrid_clock.cc:648] HybridClock initialized: now 1781894812265746 us; error 36 us; skew 500 ppm
I20260619 18:46:52.266112 4065 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:46:52 dist-test-slave-m9wg krb5kdc[4052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894812, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:46:52.268980 4065 init.cc:377] Logged in from keytab as kudu/127.3.243.126@KRBTEST.COM (short username kudu)
I20260619 18:46:52.270296 4065 webserver.cc:460] Webserver started at http://127.3.243.126:42261/ using document root <none> and password file <none>
I20260619 18:46:52.270959 4065 fs_manager.cc:359] Metadata directory not provided
I20260619 18:46:52.271072 4065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:46:52.271345 4065 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:46:52.274282 4065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894810715282-4045-0/minicluster-data/master-0/data/instance:
uuid: "4479dc6db8c24353a4dcf7b65936bbc4"
format_stamp: "Formatted at 2026-06-19 18:46:52 on dist-test-slave-m9wg"
server_key: "41d3314da0b64b41c194d9e9f49dae1a"
server_key_iv: "185a830646df5f4d8cb0a5ce72e428fa"
server_key_version: "encryptionkey@0"
I20260619 18:46:52.275012 4065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894810715282-4045-0/minicluster-data/master-0/wal/instance:
uuid: "4479dc6db8c24353a4dcf7b65936bbc4"
format_stamp: "Formatted at 2026-06-19 18:46:52 on dist-test-slave-m9wg"
server_key: "41d3314da0b64b41c194d9e9f49dae1a"
server_key_iv: "185a830646df5f4d8cb0a5ce72e428fa"
server_key_version: "encryptionkey@0"
I20260619 18:46:52.280205 4065 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.006s
I20260619 18:46:52.283448 4080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:52.284749 4065 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 18:46:52.284878 4065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894810715282-4045-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894810715282-4045-0/minicluster-data/master-0/wal
uuid: "4479dc6db8c24353a4dcf7b65936bbc4"
format_stamp: "Formatted at 2026-06-19 18:46:52 on dist-test-slave-m9wg"
server_key: "41d3314da0b64b41c194d9e9f49dae1a"
server_key_iv: "185a830646df5f4d8cb0a5ce72e428fa"
server_key_version: "encryptionkey@0"
I20260619 18:46:52.285032 4065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894810715282-4045-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894810715282-4045-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894810715282-4045-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:46:52.308254 4065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:46:52.312121 4065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:46:52.312361 4065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:46:52.329077 4065 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.126:34147
I20260619 18:46:52.329097 4131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.126:34147 every 8 connection(s)
I20260619 18:46:52.330314 4065 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
I20260619 18:46:52.333982 4132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:46:52.334679 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 4065
I20260619 18:46:52.334990 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894810715282-4045-0/minicluster-data/master-0/wal/instance
I20260619 18:46:52.337359 4045 external_mini_cluster.cc:1428] Setting key 6bf91b678a9c616bebbef3c3deb78430
I20260619 18:46:52.345614 4132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4479dc6db8c24353a4dcf7b65936bbc4: Bootstrap starting.
I20260619 18:46:52.348394 4132 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4479dc6db8c24353a4dcf7b65936bbc4: Neither blocks nor log segments found. Creating new log.
I20260619 18:46:52.349318 4132 log.cc:824] T 00000000000000000000000000000000 P 4479dc6db8c24353a4dcf7b65936bbc4: Log is configured to *not* fsync() on all Append() calls
I20260619 18:46:52.351938 4132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4479dc6db8c24353a4dcf7b65936bbc4: No bootstrap required, opened a new log
I20260619 18:46:52.361090 4132 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4479dc6db8c24353a4dcf7b65936bbc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4479dc6db8c24353a4dcf7b65936bbc4" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 34147 } }
I20260619 18:46:52.361524 4132 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4479dc6db8c24353a4dcf7b65936bbc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:46:52.361634 4132 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4479dc6db8c24353a4dcf7b65936bbc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4479dc6db8c24353a4dcf7b65936bbc4, State: Initialized, Role: FOLLOWER
I20260619 18:46:52.362320 4132 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4479dc6db8c24353a4dcf7b65936bbc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4479dc6db8c24353a4dcf7b65936bbc4" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 34147 } }
I20260619 18:46:52.362571 4132 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4479dc6db8c24353a4dcf7b65936bbc4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:46:52.362715 4132 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4479dc6db8c24353a4dcf7b65936bbc4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:46:52.362882 4132 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4479dc6db8c24353a4dcf7b65936bbc4 [term 0 FOLLOWER]: Advancing to term 1
Jun 19 18:46:52 dist-test-slave-m9wg krb5kdc[4052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894811, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.126@KRBTEST.COM
I20260619 18:46:52.365855 4132 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4479dc6db8c24353a4dcf7b65936bbc4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4479dc6db8c24353a4dcf7b65936bbc4" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 34147 } }
I20260619 18:46:52.366415 4132 leader_election.cc:302] T 00000000000000000000000000000000 P 4479dc6db8c24353a4dcf7b65936bbc4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4479dc6db8c24353a4dcf7b65936bbc4; no voters:
I20260619 18:46:52.367035 4132 leader_election.cc:288] T 00000000000000000000000000000000 P 4479dc6db8c24353a4dcf7b65936bbc4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:46:52.367115 4137 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4479dc6db8c24353a4dcf7b65936bbc4 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:46:52.368281 4137 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4479dc6db8c24353a4dcf7b65936bbc4 [term 1 LEADER]: Becoming Leader. State: Replica: 4479dc6db8c24353a4dcf7b65936bbc4, State: Running, Role: LEADER
I20260619 18:46:52.369192 4137 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4479dc6db8c24353a4dcf7b65936bbc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4479dc6db8c24353a4dcf7b65936bbc4" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 34147 } }
I20260619 18:46:52.369421 4132 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4479dc6db8c24353a4dcf7b65936bbc4 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:46:52.372633 4139 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4479dc6db8c24353a4dcf7b65936bbc4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4479dc6db8c24353a4dcf7b65936bbc4. Latest consensus state: current_term: 1 leader_uuid: "4479dc6db8c24353a4dcf7b65936bbc4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4479dc6db8c24353a4dcf7b65936bbc4" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 34147 } } }
I20260619 18:46:52.373008 4139 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4479dc6db8c24353a4dcf7b65936bbc4 [sys.catalog]: This master's current role is: LEADER
I20260619 18:46:52.373479 4138 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4479dc6db8c24353a4dcf7b65936bbc4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4479dc6db8c24353a4dcf7b65936bbc4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4479dc6db8c24353a4dcf7b65936bbc4" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 34147 } } }
I20260619 18:46:52.373771 4138 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4479dc6db8c24353a4dcf7b65936bbc4 [sys.catalog]: This master's current role is: LEADER
I20260619 18:46:52.379400 4134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:52.341646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37788 (local address 127.3.243.126:34147)
0619 18:46:52.342478 (+ 832us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:52.342492 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:52.343716 (+ 1224us) server_negotiation.cc:407] Connection header received
0619 18:46:52.351982 (+ 8266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:52.352008 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:52.352403 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:52.352717 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:52.356301 (+ 3584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:52.357627 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:52.359926 (+ 2299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:52.360617 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:52.360735 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:52.367487 (+ 6752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:52.367502 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:52.367529 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:46:52.367579 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:52.371883 (+ 4304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:52.373000 (+ 1117us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:52.373009 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:52.373018 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:52.373275 (+ 257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:52.373849 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:52.373854 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:52.373858 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:52.374610 (+ 752us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:52.377273 (+ 2663us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:52.377553 (+ 280us) server_negotiation.cc:299] Negotiation successful
0619 18:46:52.377991 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":355,"threads_started":1}
I20260619 18:46:52.385084 4145 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:46:52.389566 4145 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:46:52.397135 4145 catalog_manager.cc:1347] Generated new cluster ID: 2e654f83e732406da3eea5d08928f575
I20260619 18:46:52.397217 4145 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:46:52.406418 4145 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:46:52.407564 4145 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:46:52.412691 4145 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4479dc6db8c24353a4dcf7b65936bbc4: Generated new TSK 0
I20260619 18:46:52.413429 4145 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.243.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.65.keytab.
Entry for principal kudu/127.3.243.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.65.keytab.
I20260619 18:46:52.453771 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894810715282-4045-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894810715282-4045-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894810715282-4045-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894810715282-4045-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.65:0
--local_ip_for_outbound_sockets=127.3.243.65
--webserver_interface=127.3.243.65
--webserver_port=0
--tserver_master_addrs=127.3.243.126:34147
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.65.keytab
--principal=kudu/127.3.243.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:38057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:46:52.768388 4159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:46:52.768662 4159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:46:52.768760 4159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:46:52.776000 4159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:46:52.776140 4159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:46:52.776293 4159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.65
I20260619 18:46:52.785982 4159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:38057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894810715282-4045-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894810715282-4045-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.65.keytab
--principal=kudu/127.3.243.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894810715282-4045-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.243.65
--webserver_port=0
--tserver_master_addrs=127.3.243.126:34147
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.65
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894810715282-4045-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:46:52.787214 4159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:46:52.788569 4159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:46:52.801041 4164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:52.801082 4165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:52.800977 4167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:46:52.801854 4159 server_base.cc:1034] running on GCE node
I20260619 18:46:52.802417 4159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:46:52.803434 4159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:46:52.804616 4159 hybrid_clock.cc:648] HybridClock initialized: now 1781894812804562 us; error 51 us; skew 500 ppm
I20260619 18:46:52.804909 4159 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:46:52 dist-test-slave-m9wg krb5kdc[4052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894812, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:46:52.807799 4159 init.cc:377] Logged in from keytab as kudu/127.3.243.65@KRBTEST.COM (short username kudu)
I20260619 18:46:52.809248 4159 webserver.cc:460] Webserver started at http://127.3.243.65:36373/ using document root <none> and password file <none>
I20260619 18:46:52.809885 4159 fs_manager.cc:359] Metadata directory not provided
I20260619 18:46:52.810009 4159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:46:52.810295 4159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:46:52.813210 4159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894810715282-4045-0/minicluster-data/ts-0/data/instance:
uuid: "b3fed264222b4b58b3512c80ad7ed97d"
format_stamp: "Formatted at 2026-06-19 18:46:52 on dist-test-slave-m9wg"
server_key: "3bbeca2062c347216de4acd3bce72397"
server_key_iv: "4906751b1fd85931dddbc2139cf262b5"
server_key_version: "encryptionkey@0"
I20260619 18:46:52.813953 4159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894810715282-4045-0/minicluster-data/ts-0/wal/instance:
uuid: "b3fed264222b4b58b3512c80ad7ed97d"
format_stamp: "Formatted at 2026-06-19 18:46:52 on dist-test-slave-m9wg"
server_key: "3bbeca2062c347216de4acd3bce72397"
server_key_iv: "4906751b1fd85931dddbc2139cf262b5"
server_key_version: "encryptionkey@0"
I20260619 18:46:52.819097 4159 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 18:46:52.822393 4174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:52.823690 4159 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 18:46:52.823829 4159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894810715282-4045-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894810715282-4045-0/minicluster-data/ts-0/wal
uuid: "b3fed264222b4b58b3512c80ad7ed97d"
format_stamp: "Formatted at 2026-06-19 18:46:52 on dist-test-slave-m9wg"
server_key: "3bbeca2062c347216de4acd3bce72397"
server_key_iv: "4906751b1fd85931dddbc2139cf262b5"
server_key_version: "encryptionkey@0"
I20260619 18:46:52.824014 4159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894810715282-4045-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894810715282-4045-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894810715282-4045-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:46:52.846720 4159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:46:52.850522 4159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:46:52.850785 4159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:46:52.851682 4159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:46:52.852962 4159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:46:52.853029 4159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:52.853111 4159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:46:52.853152 4159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:52.882671 4159 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.65:45449
I20260619 18:46:52.882700 4286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.65:45449 every 8 connection(s)
I20260619 18:46:52.883867 4159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894810715282-4045-0/minicluster-data/ts-0/data/info.pb
I20260619 18:46:52.888213 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 4159
I20260619 18:46:52.888445 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894810715282-4045-0/minicluster-data/ts-0/wal/instance
I20260619 18:46:52.889173 4045 external_mini_cluster.cc:1428] Setting key 1194e00a48e96d0b47ce86f996cd09bd
Jun 19 18:46:52 dist-test-slave-m9wg krb5kdc[4052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894812, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.65@KRBTEST.COM for kudu/127.3.243.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.243.66@KRBTEST.COM; defaulting to no policy
I20260619 18:46:52.904425 4289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:52.886399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.65:46027 (local address 127.3.243.126:34147)
0619 18:46:52.886835 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:52.886841 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:52.887848 (+ 1007us) server_negotiation.cc:407] Connection header received
0619 18:46:52.890240 (+ 2392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:52.890247 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:52.890313 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:52.890419 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:52.892271 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:52.892978 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:52.894986 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:52.895506 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:52.895617 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:52.899182 (+ 3565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:52.899187 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:52.899190 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:46:52.899221 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:52.901575 (+ 2354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:52.902250 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:52.902254 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:52.902256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:52.902366 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:52.902767 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:52.902771 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:52.902773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:52.903029 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:52.903156 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:52.903828 (+ 672us) server_negotiation.cc:299] Negotiation successful
0619 18:46:52.904044 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":203,"threads_started":1}
I20260619 18:46:52.905354 4290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:52.887005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:34147 (local address 127.3.243.65:46027)
0619 18:46:52.887672 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:52.887718 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 18:46:52.890006 (+ 2288us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:46:52.890659 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:46:52.890670 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:46:52.891330 (+ 660us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:46:52.892100 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:52.892123 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:52.893115 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:52.893120 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:52.894854 (+ 1734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:52.894867 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:52.895714 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:52.895720 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:52.895903 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:52.896583 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:46:52.896604 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:46:52.898979 (+ 2375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:46:52.901700 (+ 2721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:46:52.901709 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:46:52.901726 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:46:52.902131 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:46:52.902465 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:46:52.902470 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:46:52.902474 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:46:52.902651 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:46:52.903154 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:46:52.903163 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:46:52.903490 (+ 327us) client_negotiation.cc:769] Sending connection context
0619 18:46:52.903715 (+ 225us) client_negotiation.cc:240] Negotiation successful
0619 18:46:52.904264 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":244,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.66@KRBTEST.COM" created.
I20260619 18:46:52.906839 4287 heartbeater.cc:347] Connected to a master server at 127.3.243.126:34147
I20260619 18:46:52.907171 4287 heartbeater.cc:464] Registering TS with master...
I20260619 18:46:52.908000 4287 heartbeater.cc:511] Master 127.3.243.126:34147 requested a full tablet report, sending...
I20260619 18:46:52.910344 4097 ts_manager.cc:194] Registered new tserver with Master: b3fed264222b4b58b3512c80ad7ed97d (127.3.243.65:45449)
I20260619 18:46:52.911972 4097 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.65@KRBTEST.COM'} at 127.3.243.65:46027
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.66.keytab.
Entry for principal kudu/127.3.243.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.66.keytab.
I20260619 18:46:52.925659 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894810715282-4045-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894810715282-4045-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894810715282-4045-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894810715282-4045-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.66:0
--local_ip_for_outbound_sockets=127.3.243.66
--webserver_interface=127.3.243.66
--webserver_port=0
--tserver_master_addrs=127.3.243.126:34147
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.66.keytab
--principal=kudu/127.3.243.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:38057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:46:53.235737 4293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:46:53.236016 4293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:46:53.236143 4293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:46:53.243188 4293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:46:53.243309 4293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:46:53.243468 4293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.66
I20260619 18:46:53.252898 4293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:38057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894810715282-4045-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894810715282-4045-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.66.keytab
--principal=kudu/127.3.243.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894810715282-4045-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.243.66
--webserver_port=0
--tserver_master_addrs=127.3.243.126:34147
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.66
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894810715282-4045-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:46:53.254084 4293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:46:53.255338 4293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:46:53.267339 4298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:53.267488 4299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:53.267376 4301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:46:53.268328 4293 server_base.cc:1034] running on GCE node
I20260619 18:46:53.268859 4293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:46:53.269793 4293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:46:53.270988 4293 hybrid_clock.cc:648] HybridClock initialized: now 1781894813270931 us; error 46 us; skew 500 ppm
I20260619 18:46:53.271282 4293 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:46:53 dist-test-slave-m9wg krb5kdc[4052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894813, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:46:53.274381 4293 init.cc:377] Logged in from keytab as kudu/127.3.243.66@KRBTEST.COM (short username kudu)
I20260619 18:46:53.275852 4293 webserver.cc:460] Webserver started at http://127.3.243.66:33559/ using document root <none> and password file <none>
I20260619 18:46:53.276584 4293 fs_manager.cc:359] Metadata directory not provided
I20260619 18:46:53.276701 4293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:46:53.276998 4293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:46:53.279935 4293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894810715282-4045-0/minicluster-data/ts-1/data/instance:
uuid: "b17eee5a9b8d4071affa785017da0166"
format_stamp: "Formatted at 2026-06-19 18:46:53 on dist-test-slave-m9wg"
server_key: "57e216fd21c2d7507f555ae6c9ac4d14"
server_key_iv: "f0c4d0bcb44d715c8288de2f276d5b7c"
server_key_version: "encryptionkey@0"
I20260619 18:46:53.280752 4293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894810715282-4045-0/minicluster-data/ts-1/wal/instance:
uuid: "b17eee5a9b8d4071affa785017da0166"
format_stamp: "Formatted at 2026-06-19 18:46:53 on dist-test-slave-m9wg"
server_key: "57e216fd21c2d7507f555ae6c9ac4d14"
server_key_iv: "f0c4d0bcb44d715c8288de2f276d5b7c"
server_key_version: "encryptionkey@0"
I20260619 18:46:53.285939 4293 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 18:46:53.289486 4308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:53.290786 4293 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 18:46:53.290935 4293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894810715282-4045-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894810715282-4045-0/minicluster-data/ts-1/wal
uuid: "b17eee5a9b8d4071affa785017da0166"
format_stamp: "Formatted at 2026-06-19 18:46:53 on dist-test-slave-m9wg"
server_key: "57e216fd21c2d7507f555ae6c9ac4d14"
server_key_iv: "f0c4d0bcb44d715c8288de2f276d5b7c"
server_key_version: "encryptionkey@0"
I20260619 18:46:53.291117 4293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894810715282-4045-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894810715282-4045-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894810715282-4045-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:46:53.310881 4293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:46:53.314955 4293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:46:53.315208 4293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:46:53.316181 4293 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:46:53.317529 4293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:46:53.317600 4293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:53.317687 4293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:46:53.317727 4293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:53.349815 4293 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.66:37189
I20260619 18:46:53.349860 4420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.66:37189 every 8 connection(s)
I20260619 18:46:53.350989 4293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894810715282-4045-0/minicluster-data/ts-1/data/info.pb
I20260619 18:46:53.360688 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 4293
I20260619 18:46:53.360895 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894810715282-4045-0/minicluster-data/ts-1/wal/instance
I20260619 18:46:53.361315 4045 external_mini_cluster.cc:1428] Setting key 7dc83cd70be8fd7a557f70cce386673e
Jun 19 18:46:53 dist-test-slave-m9wg krb5kdc[4052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894813, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.66@KRBTEST.COM for kudu/127.3.243.126@KRBTEST.COM
I20260619 18:46:53.370690 4289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:53.353377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.66:37655 (local address 127.3.243.126:34147)
0619 18:46:53.353557 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:53.353562 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:53.354700 (+ 1138us) server_negotiation.cc:407] Connection header received
0619 18:46:53.356573 (+ 1873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:53.356579 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:53.356663 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:53.356752 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:53.358314 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:53.359047 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:53.360817 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:53.361262 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:53.361365 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:53.365222 (+ 3857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:53.365226 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:53.365229 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:46:53.365261 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:53.367699 (+ 2438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:53.368441 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:53.368444 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:53.368447 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:53.368576 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:53.368985 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:53.368989 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:53.368991 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:53.369234 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:53.369383 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:53.370110 (+ 727us) server_negotiation.cc:299] Negotiation successful
0619 18:46:53.370335 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 18:46:53.371565 4423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:53.353807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:34147 (local address 127.3.243.66:37655)
0619 18:46:53.354534 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:53.354575 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 18:46:53.356284 (+ 1709us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:46:53.357010 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:46:53.357018 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:46:53.357550 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:46:53.358139 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:53.358157 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:53.359193 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:53.359196 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:53.360630 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:53.360646 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:53.361499 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:53.361507 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:53.361787 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:53.362461 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:46:53.362490 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:46:53.365005 (+ 2515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:46:53.367830 (+ 2825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:46:53.367840 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:46:53.367859 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:46:53.368324 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:46:53.368677 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:46:53.368682 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:46:53.368686 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:46:53.368865 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:46:53.369354 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:46:53.369362 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:46:53.369715 (+ 353us) client_negotiation.cc:769] Sending connection context
0619 18:46:53.370001 (+ 286us) client_negotiation.cc:240] Negotiation successful
0619 18:46:53.370405 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":294,"threads_started":1}
I20260619 18:46:53.372987 4421 heartbeater.cc:347] Connected to a master server at 127.3.243.126:34147
I20260619 18:46:53.373332 4421 heartbeater.cc:464] Registering TS with master...
I20260619 18:46:53.374236 4421 heartbeater.cc:511] Master 127.3.243.126:34147 requested a full tablet report, sending...
I20260619 18:46:53.375842 4097 ts_manager.cc:194] Registered new tserver with Master: b17eee5a9b8d4071affa785017da0166 (127.3.243.66:37189)
WARNING: no policy specified for kudu/127.3.243.67@KRBTEST.COM; defaulting to no policy
I20260619 18:46:53.376717 4097 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.66@KRBTEST.COM'} at 127.3.243.66:37655
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.67.keytab.
Entry for principal kudu/127.3.243.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.67.keytab.
I20260619 18:46:53.397234 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894810715282-4045-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894810715282-4045-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894810715282-4045-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894810715282-4045-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.67:0
--local_ip_for_outbound_sockets=127.3.243.67
--webserver_interface=127.3.243.67
--webserver_port=0
--tserver_master_addrs=127.3.243.126:34147
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.67.keytab
--principal=kudu/127.3.243.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:38057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:46:53.700977 4426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:46:53.701249 4426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:46:53.701406 4426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:46:53.708323 4426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:46:53.708441 4426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:46:53.708601 4426 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.67
I20260619 18:46:53.717865 4426 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:38057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894810715282-4045-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894810715282-4045-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.67.keytab
--principal=kudu/127.3.243.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894810715282-4045-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.243.67
--webserver_port=0
--tserver_master_addrs=127.3.243.126:34147
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.67
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894810715282-4045-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:46:53.719046 4426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:46:53.720302 4426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:46:53.736433 4432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:53.736470 4431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:53.736492 4434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:46:53.737263 4426 server_base.cc:1034] running on GCE node
I20260619 18:46:53.737852 4426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:46:53.738723 4426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:46:53.739910 4426 hybrid_clock.cc:648] HybridClock initialized: now 1781894813739852 us; error 54 us; skew 500 ppm
I20260619 18:46:53.740239 4426 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:46:53 dist-test-slave-m9wg krb5kdc[4052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894813, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:46:53.743218 4426 init.cc:377] Logged in from keytab as kudu/127.3.243.67@KRBTEST.COM (short username kudu)
I20260619 18:46:53.744655 4426 webserver.cc:460] Webserver started at http://127.3.243.67:35739/ using document root <none> and password file <none>
I20260619 18:46:53.745306 4426 fs_manager.cc:359] Metadata directory not provided
I20260619 18:46:53.745417 4426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:46:53.745699 4426 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:46:53.748544 4426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894810715282-4045-0/minicluster-data/ts-2/data/instance:
uuid: "34f8b22ada324d1388dc84ef92abb98d"
format_stamp: "Formatted at 2026-06-19 18:46:53 on dist-test-slave-m9wg"
server_key: "79cd1653a2e214bfac5e9c4efbaf26c4"
server_key_iv: "0ae1fdfc8377ace135943b7a1a7b1098"
server_key_version: "encryptionkey@0"
I20260619 18:46:53.749281 4426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894810715282-4045-0/minicluster-data/ts-2/wal/instance:
uuid: "34f8b22ada324d1388dc84ef92abb98d"
format_stamp: "Formatted at 2026-06-19 18:46:53 on dist-test-slave-m9wg"
server_key: "79cd1653a2e214bfac5e9c4efbaf26c4"
server_key_iv: "0ae1fdfc8377ace135943b7a1a7b1098"
server_key_version: "encryptionkey@0"
I20260619 18:46:53.754103 4426 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 18:46:53.757331 4441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:53.758564 4426 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 18:46:53.758703 4426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894810715282-4045-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894810715282-4045-0/minicluster-data/ts-2/wal
uuid: "34f8b22ada324d1388dc84ef92abb98d"
format_stamp: "Formatted at 2026-06-19 18:46:53 on dist-test-slave-m9wg"
server_key: "79cd1653a2e214bfac5e9c4efbaf26c4"
server_key_iv: "0ae1fdfc8377ace135943b7a1a7b1098"
server_key_version: "encryptionkey@0"
I20260619 18:46:53.758868 4426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894810715282-4045-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894810715282-4045-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894810715282-4045-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:46:53.773813 4426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:46:53.777726 4426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:46:53.777973 4426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:46:53.778848 4426 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:46:53.780073 4426 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:46:53.780177 4426 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:53.780285 4426 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:46:53.780364 4426 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:53.809896 4426 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.67:42905
I20260619 18:46:53.809933 4553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.67:42905 every 8 connection(s)
I20260619 18:46:53.811100 4426 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894810715282-4045-0/minicluster-data/ts-2/data/info.pb
I20260619 18:46:53.820941 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 4426
I20260619 18:46:53.821106 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894810715282-4045-0/minicluster-data/ts-2/wal/instance
I20260619 18:46:53.821646 4045 external_mini_cluster.cc:1428] Setting key 53e73c7988c83e958674b664d1850cee
Jun 19 18:46:53 dist-test-slave-m9wg krb5kdc[4052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894813, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.67@KRBTEST.COM for kudu/127.3.243.126@KRBTEST.COM
I20260619 18:46:53.829965 4289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:53.813536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.67:45575 (local address 127.3.243.126:34147)
0619 18:46:53.813685 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:53.813689 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:53.814938 (+ 1249us) server_negotiation.cc:407] Connection header received
0619 18:46:53.816858 (+ 1920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:53.816864 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:53.816942 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:53.817039 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:53.818472 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:53.819168 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:53.820746 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:53.821234 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:53.821336 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:53.824982 (+ 3646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:53.824987 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:53.824990 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:46:53.825022 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:53.827464 (+ 2442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:53.828105 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:53.828109 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:53.828111 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:53.828264 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:53.828651 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:53.828654 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:53.828656 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:53.828916 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:53.829024 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:53.829574 (+ 550us) server_negotiation.cc:299] Negotiation successful
0619 18:46:53.829728 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 18:46:53.830879 4556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:53.814027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:34147 (local address 127.3.243.67:45575)
0619 18:46:53.814757 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:53.814804 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 18:46:53.816574 (+ 1770us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:46:53.817263 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:46:53.817272 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:46:53.817758 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:46:53.818311 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:53.818328 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:53.819320 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:53.819324 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:53.820593 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:53.820608 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:53.821440 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:53.821447 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:53.821731 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:53.822382 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:46:53.822410 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:46:53.824767 (+ 2357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:46:53.827586 (+ 2819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:46:53.827592 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:46:53.827603 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:46:53.828000 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:46:53.828342 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:46:53.828344 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:46:53.828347 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:46:53.828555 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:46:53.829006 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:46:53.829011 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:46:53.829297 (+ 286us) client_negotiation.cc:769] Sending connection context
0619 18:46:53.829511 (+ 214us) client_negotiation.cc:240] Negotiation successful
0619 18:46:53.829860 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":312,"threads_started":1}
I20260619 18:46:53.832297 4554 heartbeater.cc:347] Connected to a master server at 127.3.243.126:34147
I20260619 18:46:53.832579 4554 heartbeater.cc:464] Registering TS with master...
I20260619 18:46:53.833266 4554 heartbeater.cc:511] Master 127.3.243.126:34147 requested a full tablet report, sending...
I20260619 18:46:53.834602 4097 ts_manager.cc:194] Registered new tserver with Master: 34f8b22ada324d1388dc84ef92abb98d (127.3.243.67:42905)
I20260619 18:46:53.835350 4097 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.67@KRBTEST.COM'} at 127.3.243.67:45575
I20260619 18:46:53.837374 4045 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 18:46:53 dist-test-slave-m9wg krb5kdc[4052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894813, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 18:46:53 dist-test-slave-m9wg krb5kdc[4052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894813, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.243.65@KRBTEST.COM
I20260619 18:46:53.877544 4561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:53.863736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34926 (local address 127.3.243.65:45449)
0619 18:46:53.864276 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:53.864282 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:53.864398 (+ 116us) server_negotiation.cc:407] Connection header received
0619 18:46:53.864544 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:53.864551 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:53.864767 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:53.864897 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:53.866167 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:53.867152 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:53.868415 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:53.868955 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:53.869061 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:53.872665 (+ 3604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:53.872677 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:53.872692 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:46:53.872725 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:53.875347 (+ 2622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:53.875864 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:53.875870 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:53.875873 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:53.875992 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:53.876322 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:53.876325 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:53.876327 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:53.876673 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:53.876896 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:53.877074 (+ 178us) server_negotiation.cc:299] Negotiation successful
0619 18:46:53.877260 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":270,"threads_started":1}
W20260619 18:46:53.878713 4202 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34926
Jun 19 18:46:53 dist-test-slave-m9wg krb5kdc[4052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894813, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 18:46:53 dist-test-slave-m9wg krb5kdc[4052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894813, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.65@KRBTEST.COM
I20260619 18:46:53.914857 4287 heartbeater.cc:503] Master 127.3.243.126:34147 was elected leader, sending a full tablet report...
I20260619 18:46:53.919468 4561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:53.906788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34932 (local address 127.3.243.65:45449)
0619 18:46:53.906934 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:53.906945 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:53.907298 (+ 353us) server_negotiation.cc:407] Connection header received
0619 18:46:53.907441 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:53.907445 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:53.907535 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:53.907622 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:53.908910 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:53.909604 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:53.910715 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:53.911104 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:53.911187 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:53.914686 (+ 3499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:53.914692 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:53.914695 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:46:53.914732 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:53.917565 (+ 2833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:53.918115 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:53.918119 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:53.918121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:53.918227 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:53.918489 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:53.918492 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:53.918494 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:53.918718 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:53.918874 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:53.919091 (+ 217us) server_negotiation.cc:299] Negotiation successful
0619 18:46:53.919241 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 18:46:53.920830 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 4159
I20260619 18:46:53.932191 4561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:53.921708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34938 (local address 127.3.243.65:45449)
0619 18:46:53.921894 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:53.921899 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:53.922049 (+ 150us) server_negotiation.cc:407] Connection header received
0619 18:46:53.922150 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:53.922154 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:53.922204 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:53.922273 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:53.923370 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:53.924036 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:53.925181 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:53.925569 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:53.925653 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:53.927816 (+ 2163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:53.927820 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:53.927823 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:46:53.927849 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:53.930099 (+ 2250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:53.930620 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:53.930623 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:53.930625 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:53.930729 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:53.931076 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:53.931078 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:53.931080 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:53.931318 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:53.931505 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:53.931719 (+ 214us) server_negotiation.cc:299] Negotiation successful
0619 18:46:53.931856 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 18:46:53.932840 4282 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:46:54.012773 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 4159
I20260619 18:46:54.025171 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 4293
Jun 19 18:46:54 dist-test-slave-m9wg krb5kdc[4052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894813, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.66@KRBTEST.COM
I20260619 18:46:54.039470 4568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:54.025932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56924 (local address 127.3.243.66:37189)
0619 18:46:54.026500 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:54.026507 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:54.026532 (+ 25us) server_negotiation.cc:407] Connection header received
0619 18:46:54.026596 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:54.026602 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:54.026801 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:54.026927 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:54.027945 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:54.028977 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:54.030110 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:54.030631 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:54.030719 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:54.034547 (+ 3828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:54.034558 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:54.034572 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:46:54.034607 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:54.037184 (+ 2577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:54.037796 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:54.037802 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:54.037805 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:54.037916 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:54.038215 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:54.038218 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:54.038220 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:54.038520 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:54.038734 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:54.039003 (+ 269us) server_negotiation.cc:299] Negotiation successful
0619 18:46:54.039195 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":284,"threads_started":1}
I20260619 18:46:54.040747 4416 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:46:54.122359 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 4293
I20260619 18:46:54.134320 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 4426
Jun 19 18:46:54 dist-test-slave-m9wg krb5kdc[4052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894813, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.67@KRBTEST.COM
I20260619 18:46:54.149542 4570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:54.135261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48898 (local address 127.3.243.67:42905)
0619 18:46:54.135842 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:54.135849 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:54.135877 (+ 28us) server_negotiation.cc:407] Connection header received
0619 18:46:54.135944 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:54.135950 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:54.136182 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:54.136335 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:54.137576 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:54.138573 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:54.139808 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:54.140429 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:54.140537 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:54.144545 (+ 4008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:54.144556 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:54.144570 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:46:54.144600 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:54.147224 (+ 2624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:54.147806 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:54.147813 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:54.147816 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:54.147951 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:54.148255 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:54.148259 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:54.148262 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:54.148587 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:54.148812 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:54.149073 (+ 261us) server_negotiation.cc:299] Negotiation successful
0619 18:46:54.149263 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":303,"threads_started":1}
I20260619 18:46:54.150774 4549 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:46:54.230918 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 4426
I20260619 18:46:54.243070 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 4065
I20260619 18:46:54.244223 4127 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:46:54.343506 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 4065
2026-06-19T18:46:54Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3592 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894810715282-4045-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:46:54 dist-test-slave-m9wg krb5kdc[4575](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:46:54 dist-test-slave-m9wg krb5kdc[4575](info): set up 2 sockets
Jun 19 18:46:54 dist-test-slave-m9wg krb5kdc[4575](info): commencing operation
krb5kdc: starting...
W20260619 18:46:55.323872 4045 mini_kdc.cc:121] Time spent starting KDC: real 0.944s user 0.002s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894810715282-4045-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894810715282-4045-0/krb5kdc/test-user.keytab.
Jun 19 18:46:55 dist-test-slave-m9wg krb5kdc[4575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894815, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:46:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:46:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.243.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab.
Entry for principal kudu/127.3.243.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab.
I20260619 18:46:55.471921 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894810715282-4045-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894810715282-4045-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894810715282-4045-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.126:37415
--webserver_interface=127.3.243.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab
--principal=kudu/127.3.243.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:45365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.126:37415
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:46:55.771485 4588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:46:55.771775 4588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:46:55.771899 4588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:46:55.779075 4588 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:46:55.779189 4588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:46:55.779264 4588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:46:55.779337 4588 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:46:55.779397 4588 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:46:55.789004 4588 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:45365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894810715282-4045-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894810715282-4045-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.243.126:37415
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab
--principal=kudu/127.3.243.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.126:37415
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.243.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894810715282-4045-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:46:55.790304 4588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:46:55.791599 4588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:46:55.802508 4593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:55.802539 4596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:55.802474 4594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:46:55.803653 4588 server_base.cc:1034] running on GCE node
I20260619 18:46:55.804399 4588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:46:55.805680 4588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:46:55.806890 4588 hybrid_clock.cc:648] HybridClock initialized: now 1781894815806833 us; error 45 us; skew 500 ppm
I20260619 18:46:55.807228 4588 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:46:55 dist-test-slave-m9wg krb5kdc[4575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894815, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:46:55.810179 4588 init.cc:377] Logged in from keytab as kudu/127.3.243.126@KRBTEST.COM (short username kudu)
I20260619 18:46:55.811648 4588 webserver.cc:460] Webserver started at http://127.3.243.126:34967/ using document root <none> and password file <none>
I20260619 18:46:55.812319 4588 fs_manager.cc:359] Metadata directory not provided
I20260619 18:46:55.812444 4588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:46:55.812734 4588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:46:55.815601 4588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894810715282-4045-0/minicluster-data/master-0/data/instance:
uuid: "68dbe1747f44435d809cd9d60762ae2b"
format_stamp: "Formatted at 2026-06-19 18:46:55 on dist-test-slave-m9wg"
server_key: "c279cbbbb50149c6b7e2e665282aaee2"
server_key_iv: "35f148bf98598f778d104b2e1633fcef"
server_key_version: "encryptionkey@0"
I20260619 18:46:55.816393 4588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894810715282-4045-0/minicluster-data/master-0/wal/instance:
uuid: "68dbe1747f44435d809cd9d60762ae2b"
format_stamp: "Formatted at 2026-06-19 18:46:55 on dist-test-slave-m9wg"
server_key: "c279cbbbb50149c6b7e2e665282aaee2"
server_key_iv: "35f148bf98598f778d104b2e1633fcef"
server_key_version: "encryptionkey@0"
I20260619 18:46:55.821300 4588 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 18:46:55.824571 4603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:55.825822 4588 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 18:46:55.825985 4588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894810715282-4045-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894810715282-4045-0/minicluster-data/master-0/wal
uuid: "68dbe1747f44435d809cd9d60762ae2b"
format_stamp: "Formatted at 2026-06-19 18:46:55 on dist-test-slave-m9wg"
server_key: "c279cbbbb50149c6b7e2e665282aaee2"
server_key_iv: "35f148bf98598f778d104b2e1633fcef"
server_key_version: "encryptionkey@0"
I20260619 18:46:55.826160 4588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894810715282-4045-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894810715282-4045-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894810715282-4045-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:46:55.880280 4588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:46:55.884574 4588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:46:55.884816 4588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:46:55.901095 4588 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.126:37415
I20260619 18:46:55.901119 4654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.126:37415 every 8 connection(s)
I20260619 18:46:55.902335 4588 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
I20260619 18:46:55.905117 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 4588
I20260619 18:46:55.905295 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894810715282-4045-0/minicluster-data/master-0/wal/instance
I20260619 18:46:55.905874 4045 external_mini_cluster.cc:1428] Setting key e853e1919f2b63ec9dc8cc4f020084c8
I20260619 18:46:55.905972 4655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:46:55.919459 4655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68dbe1747f44435d809cd9d60762ae2b: Bootstrap starting.
Jun 19 18:46:55 dist-test-slave-m9wg krb5kdc[4575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894815, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.126@KRBTEST.COM
I20260619 18:46:55.923050 4655 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 68dbe1747f44435d809cd9d60762ae2b: Neither blocks nor log segments found. Creating new log.
I20260619 18:46:55.924330 4655 log.cc:824] T 00000000000000000000000000000000 P 68dbe1747f44435d809cd9d60762ae2b: Log is configured to *not* fsync() on all Append() calls
I20260619 18:46:55.926955 4655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68dbe1747f44435d809cd9d60762ae2b: No bootstrap required, opened a new log
I20260619 18:46:55.928205 4658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:55.907712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37874 (local address 127.3.243.126:37415)
0619 18:46:55.908569 (+ 857us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:55.908583 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:55.908634 (+ 51us) server_negotiation.cc:407] Connection header received
0619 18:46:55.910587 (+ 1953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:55.910613 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:55.911112 (+ 499us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:55.911558 (+ 446us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:55.912792 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:55.914750 (+ 1958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:55.915906 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:55.916997 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:55.917178 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:55.920815 (+ 3637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:55.920825 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:55.920847 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:46:55.920908 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:55.924544 (+ 3636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:55.925161 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:55.925167 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:55.925173 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:55.925335 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:55.925671 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:55.925675 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:55.925677 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:55.926146 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:55.926409 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:55.926858 (+ 449us) server_negotiation.cc:299] Negotiation successful
0619 18:46:55.927246 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":365,"threads_started":1}
I20260619 18:46:55.935180 4655 raft_consensus.cc:348] T 00000000000000000000000000000000 P 68dbe1747f44435d809cd9d60762ae2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68dbe1747f44435d809cd9d60762ae2b" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 37415 } }
I20260619 18:46:55.935568 4655 raft_consensus.cc:374] T 00000000000000000000000000000000 P 68dbe1747f44435d809cd9d60762ae2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:46:55.935654 4655 raft_consensus.cc:729] T 00000000000000000000000000000000 P 68dbe1747f44435d809cd9d60762ae2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68dbe1747f44435d809cd9d60762ae2b, State: Initialized, Role: FOLLOWER
I20260619 18:46:55.936237 4655 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68dbe1747f44435d809cd9d60762ae2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68dbe1747f44435d809cd9d60762ae2b" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 37415 } }
I20260619 18:46:55.936479 4655 raft_consensus.cc:388] T 00000000000000000000000000000000 P 68dbe1747f44435d809cd9d60762ae2b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:46:55.936605 4655 raft_consensus.cc:482] T 00000000000000000000000000000000 P 68dbe1747f44435d809cd9d60762ae2b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:46:55.936748 4655 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 68dbe1747f44435d809cd9d60762ae2b [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:46:55.938761 4655 raft_consensus.cc:504] T 00000000000000000000000000000000 P 68dbe1747f44435d809cd9d60762ae2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68dbe1747f44435d809cd9d60762ae2b" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 37415 } }
I20260619 18:46:55.939284 4655 leader_election.cc:302] T 00000000000000000000000000000000 P 68dbe1747f44435d809cd9d60762ae2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 68dbe1747f44435d809cd9d60762ae2b; no voters:
I20260619 18:46:55.939823 4655 leader_election.cc:288] T 00000000000000000000000000000000 P 68dbe1747f44435d809cd9d60762ae2b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:46:55.939945 4660 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 68dbe1747f44435d809cd9d60762ae2b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:46:55.940554 4660 raft_consensus.cc:686] T 00000000000000000000000000000000 P 68dbe1747f44435d809cd9d60762ae2b [term 1 LEADER]: Becoming Leader. State: Replica: 68dbe1747f44435d809cd9d60762ae2b, State: Running, Role: LEADER
I20260619 18:46:55.941071 4660 consensus_queue.cc:237] T 00000000000000000000000000000000 P 68dbe1747f44435d809cd9d60762ae2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68dbe1747f44435d809cd9d60762ae2b" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 37415 } }
I20260619 18:46:55.941668 4655 sys_catalog.cc:564] T 00000000000000000000000000000000 P 68dbe1747f44435d809cd9d60762ae2b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:46:55.943679 4661 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68dbe1747f44435d809cd9d60762ae2b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "68dbe1747f44435d809cd9d60762ae2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68dbe1747f44435d809cd9d60762ae2b" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 37415 } } }
I20260619 18:46:55.943727 4662 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68dbe1747f44435d809cd9d60762ae2b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 68dbe1747f44435d809cd9d60762ae2b. Latest consensus state: current_term: 1 leader_uuid: "68dbe1747f44435d809cd9d60762ae2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68dbe1747f44435d809cd9d60762ae2b" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 37415 } } }
I20260619 18:46:55.944010 4661 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68dbe1747f44435d809cd9d60762ae2b [sys.catalog]: This master's current role is: LEADER
I20260619 18:46:55.944080 4662 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68dbe1747f44435d809cd9d60762ae2b [sys.catalog]: This master's current role is: LEADER
I20260619 18:46:55.944792 4667 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:46:55.950274 4667 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:46:55.959712 4667 catalog_manager.cc:1347] Generated new cluster ID: c1b4fd00b0754e91b7c26456385b451a
I20260619 18:46:55.959803 4667 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:46:55.972260 4667 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:46:55.973510 4667 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:46:55.986155 4667 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 68dbe1747f44435d809cd9d60762ae2b: Generated new TSK 0
I20260619 18:46:55.986929 4667 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.243.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.65.keytab.
Entry for principal kudu/127.3.243.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.65.keytab.
I20260619 18:46:56.027631 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894810715282-4045-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894810715282-4045-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894810715282-4045-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894810715282-4045-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.65:0
--local_ip_for_outbound_sockets=127.3.243.65
--webserver_interface=127.3.243.65
--webserver_port=0
--tserver_master_addrs=127.3.243.126:37415
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.65.keytab
--principal=kudu/127.3.243.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:45365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:46:56.330336 4681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:46:56.330610 4681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:46:56.330742 4681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:46:56.337930 4681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:46:56.338035 4681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:46:56.338195 4681 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.65
I20260619 18:46:56.347788 4681 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:45365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894810715282-4045-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894810715282-4045-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.65.keytab
--principal=kudu/127.3.243.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894810715282-4045-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.243.65
--webserver_port=0
--tserver_master_addrs=127.3.243.126:37415
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.65
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894810715282-4045-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:46:56.349035 4681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:46:56.350373 4681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:46:56.363083 4686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:56.363111 4689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:56.363276 4687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:46:56.363684 4681 server_base.cc:1034] running on GCE node
I20260619 18:46:56.364333 4681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:46:56.365223 4681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:46:56.366436 4681 hybrid_clock.cc:648] HybridClock initialized: now 1781894816366360 us; error 36 us; skew 500 ppm
I20260619 18:46:56.366719 4681 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:46:56 dist-test-slave-m9wg krb5kdc[4575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894816, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:46:56.369621 4681 init.cc:377] Logged in from keytab as kudu/127.3.243.65@KRBTEST.COM (short username kudu)
I20260619 18:46:56.370939 4681 webserver.cc:460] Webserver started at http://127.3.243.65:43493/ using document root <none> and password file <none>
I20260619 18:46:56.371568 4681 fs_manager.cc:359] Metadata directory not provided
I20260619 18:46:56.371680 4681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:46:56.371956 4681 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:46:56.374742 4681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894810715282-4045-0/minicluster-data/ts-0/data/instance:
uuid: "316db71149464366baf681833b019e6a"
format_stamp: "Formatted at 2026-06-19 18:46:56 on dist-test-slave-m9wg"
server_key: "5714edc1834236416ace8813cc18d59c"
server_key_iv: "3710cc6544464eadc21498d410c68d7a"
server_key_version: "encryptionkey@0"
I20260619 18:46:56.375491 4681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894810715282-4045-0/minicluster-data/ts-0/wal/instance:
uuid: "316db71149464366baf681833b019e6a"
format_stamp: "Formatted at 2026-06-19 18:46:56 on dist-test-slave-m9wg"
server_key: "5714edc1834236416ace8813cc18d59c"
server_key_iv: "3710cc6544464eadc21498d410c68d7a"
server_key_version: "encryptionkey@0"
I20260619 18:46:56.380221 4681 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.005s
I20260619 18:46:56.383325 4696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:56.384618 4681 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:46:56.384778 4681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894810715282-4045-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894810715282-4045-0/minicluster-data/ts-0/wal
uuid: "316db71149464366baf681833b019e6a"
format_stamp: "Formatted at 2026-06-19 18:46:56 on dist-test-slave-m9wg"
server_key: "5714edc1834236416ace8813cc18d59c"
server_key_iv: "3710cc6544464eadc21498d410c68d7a"
server_key_version: "encryptionkey@0"
I20260619 18:46:56.384981 4681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894810715282-4045-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894810715282-4045-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894810715282-4045-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:46:56.400238 4681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:46:56.404055 4681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:46:56.404340 4681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:46:56.405270 4681 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:46:56.406594 4681 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:46:56.406668 4681 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:56.406781 4681 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:46:56.406832 4681 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:56.435565 4681 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.65:32987
I20260619 18:46:56.435599 4808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.65:32987 every 8 connection(s)
I20260619 18:46:56.436797 4681 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894810715282-4045-0/minicluster-data/ts-0/data/info.pb
I20260619 18:46:56.442868 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 4681
I20260619 18:46:56.443066 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894810715282-4045-0/minicluster-data/ts-0/wal/instance
I20260619 18:46:56.443625 4045 external_mini_cluster.cc:1428] Setting key 7d3ec7eba9681c6b40e4a239e632ffb6
Jun 19 18:46:56 dist-test-slave-m9wg krb5kdc[4575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894816, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.65@KRBTEST.COM for kudu/127.3.243.126@KRBTEST.COM
I20260619 18:46:56.456264 4811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:56.439064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.65:45017 (local address 127.3.243.126:37415)
0619 18:46:56.439506 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:56.439512 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:56.440459 (+ 947us) server_negotiation.cc:407] Connection header received
0619 18:46:56.442168 (+ 1709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:56.442172 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:56.442238 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:56.442333 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:56.443911 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:56.444823 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:56.446548 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:56.447162 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:56.447289 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:56.450439 (+ 3150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:56.450444 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:56.450448 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:46:56.450479 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:56.453335 (+ 2856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:56.453990 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:56.453995 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:56.453999 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:56.454140 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:56.454491 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:56.454495 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:56.454498 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:56.454828 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:56.454984 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:56.455634 (+ 650us) server_negotiation.cc:299] Negotiation successful
0619 18:46:56.455867 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":200,"threads_started":1}
I20260619 18:46:56.457068 4812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:56.439597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:37415 (local address 127.3.243.65:45017)
0619 18:46:56.440282 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:56.440328 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 18:46:56.441941 (+ 1613us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:46:56.442554 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:46:56.442564 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:46:56.443201 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:46:56.443757 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:56.443775 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:56.444976 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:56.444981 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:56.446395 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:56.446408 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:56.447357 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:56.447362 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:56.447575 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:56.448202 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:46:56.448224 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:46:56.450222 (+ 1998us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:46:56.453475 (+ 3253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:46:56.453481 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:46:56.453492 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:46:56.453891 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:46:56.454231 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:46:56.454234 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:46:56.454237 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:46:56.454400 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:46:56.454982 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:46:56.454989 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:46:56.455286 (+ 297us) client_negotiation.cc:769] Sending connection context
0619 18:46:56.455507 (+ 221us) client_negotiation.cc:240] Negotiation successful
0619 18:46:56.455857 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":245,"threads_started":1}
I20260619 18:46:56.458650 4809 heartbeater.cc:347] Connected to a master server at 127.3.243.126:37415
WARNING: no policy specified for kudu/127.3.243.66@KRBTEST.COM; defaulting to no policy
I20260619 18:46:56.458961 4809 heartbeater.cc:464] Registering TS with master...
I20260619 18:46:56.459647 4809 heartbeater.cc:511] Master 127.3.243.126:37415 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.66@KRBTEST.COM" created.
I20260619 18:46:56.461862 4620 ts_manager.cc:194] Registered new tserver with Master: 316db71149464366baf681833b019e6a (127.3.243.65:32987)
I20260619 18:46:56.463742 4620 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.65@KRBTEST.COM'} at 127.3.243.65:45017
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.66.keytab.
Entry for principal kudu/127.3.243.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.66.keytab.
I20260619 18:46:56.480713 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894810715282-4045-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894810715282-4045-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894810715282-4045-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894810715282-4045-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.66:0
--local_ip_for_outbound_sockets=127.3.243.66
--webserver_interface=127.3.243.66
--webserver_port=0
--tserver_master_addrs=127.3.243.126:37415
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.66.keytab
--principal=kudu/127.3.243.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:45365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:46:56.780925 4815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:46:56.781183 4815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:46:56.781313 4815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:46:56.788415 4815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:46:56.788511 4815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:46:56.788658 4815 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.66
I20260619 18:46:56.798194 4815 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:45365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894810715282-4045-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894810715282-4045-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.66.keytab
--principal=kudu/127.3.243.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894810715282-4045-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.243.66
--webserver_port=0
--tserver_master_addrs=127.3.243.126:37415
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.66
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894810715282-4045-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:46:56.799388 4815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:46:56.800675 4815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:46:56.812587 4821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:56.812774 4820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:56.812558 4823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:46:56.813360 4815 server_base.cc:1034] running on GCE node
I20260619 18:46:56.813937 4815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:46:56.814816 4815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:46:56.816012 4815 hybrid_clock.cc:648] HybridClock initialized: now 1781894816815966 us; error 39 us; skew 500 ppm
I20260619 18:46:56.816354 4815 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:46:56 dist-test-slave-m9wg krb5kdc[4575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894816, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:46:56.819229 4815 init.cc:377] Logged in from keytab as kudu/127.3.243.66@KRBTEST.COM (short username kudu)
I20260619 18:46:56.820670 4815 webserver.cc:460] Webserver started at http://127.3.243.66:41265/ using document root <none> and password file <none>
I20260619 18:46:56.821300 4815 fs_manager.cc:359] Metadata directory not provided
I20260619 18:46:56.821408 4815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:46:56.821691 4815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:46:56.824479 4815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894810715282-4045-0/minicluster-data/ts-1/data/instance:
uuid: "6c1a2c24714b47678fef1d20ed8bce0e"
format_stamp: "Formatted at 2026-06-19 18:46:56 on dist-test-slave-m9wg"
server_key: "788e834ea849ed8e18df4f3efd22edab"
server_key_iv: "1cbefd34b17eed9ae3fdac7658a4c33f"
server_key_version: "encryptionkey@0"
I20260619 18:46:56.825207 4815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894810715282-4045-0/minicluster-data/ts-1/wal/instance:
uuid: "6c1a2c24714b47678fef1d20ed8bce0e"
format_stamp: "Formatted at 2026-06-19 18:46:56 on dist-test-slave-m9wg"
server_key: "788e834ea849ed8e18df4f3efd22edab"
server_key_iv: "1cbefd34b17eed9ae3fdac7658a4c33f"
server_key_version: "encryptionkey@0"
I20260619 18:46:56.830057 4815 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 18:46:56.833244 4830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:56.834436 4815 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.003s
I20260619 18:46:56.834573 4815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894810715282-4045-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894810715282-4045-0/minicluster-data/ts-1/wal
uuid: "6c1a2c24714b47678fef1d20ed8bce0e"
format_stamp: "Formatted at 2026-06-19 18:46:56 on dist-test-slave-m9wg"
server_key: "788e834ea849ed8e18df4f3efd22edab"
server_key_iv: "1cbefd34b17eed9ae3fdac7658a4c33f"
server_key_version: "encryptionkey@0"
I20260619 18:46:56.834729 4815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894810715282-4045-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894810715282-4045-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894810715282-4045-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:46:56.846536 4815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:46:56.850051 4815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:46:56.850288 4815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:46:56.851161 4815 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:46:56.852437 4815 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:46:56.852509 4815 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:56.852622 4815 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:46:56.852671 4815 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:56.881022 4815 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.66:42479
I20260619 18:46:56.881072 4942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.66:42479 every 8 connection(s)
I20260619 18:46:56.882187 4815 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894810715282-4045-0/minicluster-data/ts-1/data/info.pb
I20260619 18:46:56.884984 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 4815
I20260619 18:46:56.885150 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894810715282-4045-0/minicluster-data/ts-1/wal/instance
I20260619 18:46:56.885537 4045 external_mini_cluster.cc:1428] Setting key 52a4a9648263c7a432f56514d708c781
Jun 19 18:46:56 dist-test-slave-m9wg krb5kdc[4575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894816, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.66@KRBTEST.COM for kudu/127.3.243.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.243.67@KRBTEST.COM; defaulting to no policy
I20260619 18:46:56.902479 4811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:56.884614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.66:41649 (local address 127.3.243.126:37415)
0619 18:46:56.884821 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:56.884827 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:56.886218 (+ 1391us) server_negotiation.cc:407] Connection header received
0619 18:46:56.888229 (+ 2011us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:56.888235 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:56.888308 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:56.888425 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:56.890291 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:56.890976 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:56.892846 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:56.893277 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:56.893382 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:56.896709 (+ 3327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:56.896714 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:56.896717 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:46:56.896748 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:56.899146 (+ 2398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:56.899976 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:56.899980 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:56.899982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:56.900106 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:56.900616 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:56.900621 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:56.900624 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:56.900965 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:56.901115 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:56.901858 (+ 743us) server_negotiation.cc:299] Negotiation successful
0619 18:46:56.902120 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 18:46:56.903080 4945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:56.885147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:37415 (local address 127.3.243.66:41649)
0619 18:46:56.886055 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:56.886104 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 18:46:56.887913 (+ 1809us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:46:56.888691 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:46:56.888703 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:46:56.889404 (+ 701us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:46:56.890133 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:56.890158 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:56.891115 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:56.891120 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:56.892711 (+ 1591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:56.892727 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:56.893485 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:56.893492 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:56.893692 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:56.894309 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:46:56.894329 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:46:56.896521 (+ 2192us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:46:56.899277 (+ 2756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:46:56.899286 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:46:56.899301 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:46:56.899872 (+ 571us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:46:56.900232 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:46:56.900237 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:46:56.900241 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:46:56.900493 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:46:56.901099 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:46:56.901107 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:46:56.901513 (+ 406us) client_negotiation.cc:769] Sending connection context
0619 18:46:56.901729 (+ 216us) client_negotiation.cc:240] Negotiation successful
0619 18:46:56.902074 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":652,"thread_start_us":298,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.67@KRBTEST.COM" created.
I20260619 18:46:56.904696 4943 heartbeater.cc:347] Connected to a master server at 127.3.243.126:37415
I20260619 18:46:56.904971 4943 heartbeater.cc:464] Registering TS with master...
I20260619 18:46:56.905633 4943 heartbeater.cc:511] Master 127.3.243.126:37415 requested a full tablet report, sending...
I20260619 18:46:56.906900 4620 ts_manager.cc:194] Registered new tserver with Master: 6c1a2c24714b47678fef1d20ed8bce0e (127.3.243.66:42479)
I20260619 18:46:56.907701 4620 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.66@KRBTEST.COM'} at 127.3.243.66:41649
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.67.keytab.
Entry for principal kudu/127.3.243.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.67.keytab.
I20260619 18:46:56.926546 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894810715282-4045-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894810715282-4045-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894810715282-4045-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894810715282-4045-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.67:0
--local_ip_for_outbound_sockets=127.3.243.67
--webserver_interface=127.3.243.67
--webserver_port=0
--tserver_master_addrs=127.3.243.126:37415
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.67.keytab
--principal=kudu/127.3.243.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:45365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:46:57.225940 4948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:46:57.226209 4948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:46:57.226344 4948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:46:57.233541 4948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:46:57.233654 4948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:46:57.233799 4948 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.67
I20260619 18:46:57.243384 4948 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:45365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894810715282-4045-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894810715282-4045-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.67.keytab
--principal=kudu/127.3.243.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894810715282-4045-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.243.67
--webserver_port=0
--tserver_master_addrs=127.3.243.126:37415
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.67
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894810715282-4045-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:46:57.244639 4948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:46:57.245873 4948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:46:57.257880 4956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:57.257913 4953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:57.257942 4954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:46:57.258455 4948 server_base.cc:1034] running on GCE node
I20260619 18:46:57.258980 4948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:46:57.259809 4948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:46:57.260988 4948 hybrid_clock.cc:648] HybridClock initialized: now 1781894817260947 us; error 33 us; skew 500 ppm
I20260619 18:46:57.261258 4948 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:46:57 dist-test-slave-m9wg krb5kdc[4575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894817, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:46:57.264106 4948 init.cc:377] Logged in from keytab as kudu/127.3.243.67@KRBTEST.COM (short username kudu)
I20260619 18:46:57.265492 4948 webserver.cc:460] Webserver started at http://127.3.243.67:35237/ using document root <none> and password file <none>
I20260619 18:46:57.266134 4948 fs_manager.cc:359] Metadata directory not provided
I20260619 18:46:57.266235 4948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:46:57.266517 4948 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:46:57.269347 4948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894810715282-4045-0/minicluster-data/ts-2/data/instance:
uuid: "326b688a705a47a6b34539130d5540b0"
format_stamp: "Formatted at 2026-06-19 18:46:57 on dist-test-slave-m9wg"
server_key: "b18439fd936a61ccd47775421fb46bfe"
server_key_iv: "1d4c9b83d51e62299793888a87f4f488"
server_key_version: "encryptionkey@0"
I20260619 18:46:57.270092 4948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894810715282-4045-0/minicluster-data/ts-2/wal/instance:
uuid: "326b688a705a47a6b34539130d5540b0"
format_stamp: "Formatted at 2026-06-19 18:46:57 on dist-test-slave-m9wg"
server_key: "b18439fd936a61ccd47775421fb46bfe"
server_key_iv: "1d4c9b83d51e62299793888a87f4f488"
server_key_version: "encryptionkey@0"
I20260619 18:46:57.274935 4948 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 18:46:57.278198 4963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:57.279695 4948 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 18:46:57.279847 4948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894810715282-4045-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894810715282-4045-0/minicluster-data/ts-2/wal
uuid: "326b688a705a47a6b34539130d5540b0"
format_stamp: "Formatted at 2026-06-19 18:46:57 on dist-test-slave-m9wg"
server_key: "b18439fd936a61ccd47775421fb46bfe"
server_key_iv: "1d4c9b83d51e62299793888a87f4f488"
server_key_version: "encryptionkey@0"
I20260619 18:46:57.280030 4948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894810715282-4045-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894810715282-4045-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894810715282-4045-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:46:57.303905 4948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:46:57.308036 4948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:46:57.308326 4948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:46:57.309597 4948 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:46:57.310869 4948 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:46:57.310946 4948 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:57.311062 4948 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:46:57.311113 4948 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:57.340963 4948 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.67:42415
I20260619 18:46:57.341032 5075 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.67:42415 every 8 connection(s)
I20260619 18:46:57.342214 4948 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894810715282-4045-0/minicluster-data/ts-2/data/info.pb
I20260619 18:46:57.350056 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 4948
I20260619 18:46:57.350240 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894810715282-4045-0/minicluster-data/ts-2/wal/instance
I20260619 18:46:57.350791 4045 external_mini_cluster.cc:1428] Setting key 9bae13d7b9404be6fe5d5f68359e41d4
Jun 19 18:46:57 dist-test-slave-m9wg krb5kdc[4575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894817, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.67@KRBTEST.COM for kudu/127.3.243.126@KRBTEST.COM
I20260619 18:46:57.360838 4811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:57.344652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.67:53069 (local address 127.3.243.126:37415)
0619 18:46:57.344849 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:57.344855 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:57.346064 (+ 1209us) server_negotiation.cc:407] Connection header received
0619 18:46:57.347793 (+ 1729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:57.347797 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:57.347866 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:57.347962 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:57.349369 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:57.350110 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:57.352038 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:57.352616 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:57.352727 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:57.355800 (+ 3073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:57.355805 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:57.355808 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:46:57.355840 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:57.358118 (+ 2278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:57.358755 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:57.358760 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:57.358764 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:57.358883 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:57.359279 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:57.359282 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:57.359285 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:57.359551 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:57.359676 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:57.360301 (+ 625us) server_negotiation.cc:299] Negotiation successful
0619 18:46:57.360520 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 18:46:57.361726 5078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:57.345206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:37415 (local address 127.3.243.67:53069)
0619 18:46:57.345888 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:57.345931 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 18:46:57.347549 (+ 1618us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:46:57.348166 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:46:57.348178 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:46:57.348703 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:46:57.349236 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:57.349253 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:57.350285 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:57.350291 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:57.351906 (+ 1615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:57.351918 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:57.352823 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:57.352830 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:57.353058 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:57.353636 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:46:57.353657 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:46:57.355608 (+ 1951us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:46:57.358240 (+ 2632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:46:57.358249 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:46:57.358265 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:46:57.358651 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:46:57.358980 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:46:57.358985 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:46:57.358988 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:46:57.359156 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:46:57.359674 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:46:57.359681 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:46:57.359963 (+ 282us) client_negotiation.cc:769] Sending connection context
0619 18:46:57.360213 (+ 250us) client_negotiation.cc:240] Negotiation successful
0619 18:46:57.360699 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":261,"threads_started":1}
I20260619 18:46:57.362891 5076 heartbeater.cc:347] Connected to a master server at 127.3.243.126:37415
I20260619 18:46:57.363132 5076 heartbeater.cc:464] Registering TS with master...
I20260619 18:46:57.363750 5076 heartbeater.cc:511] Master 127.3.243.126:37415 requested a full tablet report, sending...
I20260619 18:46:57.364987 4620 ts_manager.cc:194] Registered new tserver with Master: 326b688a705a47a6b34539130d5540b0 (127.3.243.67:42415)
I20260619 18:46:57.365873 4620 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.67@KRBTEST.COM'} at 127.3.243.67:53069
I20260619 18:46:57.365952 4045 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 18:46:57.382798 4811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:57.371763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37890 (local address 127.3.243.126:37415)
0619 18:46:57.371897 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:57.371901 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:57.372386 (+ 485us) server_negotiation.cc:407] Connection header received
0619 18:46:57.372518 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:57.372521 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:57.372577 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:57.372651 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:57.373955 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:57.374642 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:57.375838 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:57.376260 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:57.376363 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:57.378405 (+ 2042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:57.378410 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:57.378415 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:46:57.378443 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:57.380822 (+ 2379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:57.381334 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:57.381338 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:57.381340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:57.381445 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:57.381698 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:57.381701 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:57.381703 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:57.381991 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:57.382157 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:57.382409 (+ 252us) server_negotiation.cc:299] Negotiation successful
0619 18:46:57.382554 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 18:46:57.388865 4620 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37890:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:46:57.395521 4620 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:46:57.420116 5089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:57.407664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.67:42415 (local address 127.0.0.1:60604)
0619 18:46:57.410513 (+ 2849us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:57.410547 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 18:46:57.410725 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:46:57.411703 (+ 978us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:46:57.411712 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:46:57.411748 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:46:57.412125 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:57.412137 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:57.414257 (+ 2120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:57.414263 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:57.416841 (+ 2578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:57.416858 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:57.419093 (+ 2235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:57.419100 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:57.419314 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:57.419368 (+ 54us) client_negotiation.cc:769] Sending connection context
0619 18:46:57.419503 (+ 135us) client_negotiation.cc:240] Negotiation successful
0619 18:46:57.419696 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2588,"thread_start_us":3249,"threads_started":1}
I20260619 18:46:57.420430 5091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:57.408433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48462 (local address 127.3.243.66:42479)
0619 18:46:57.409436 (+ 1003us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:57.409446 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:57.411268 (+ 1822us) server_negotiation.cc:407] Connection header received
0619 18:46:57.411547 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:57.411555 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:57.411773 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:57.411936 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:46:57.412843 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:57.414490 (+ 1647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:57.417508 (+ 3018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:57.418979 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:57.419085 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:57.419261 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:57.419841 (+ 580us) server_negotiation.cc:299] Negotiation successful
0619 18:46:57.420042 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":845,"thread_start_us":735,"threads_started":1}
I20260619 18:46:57.421291 5090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:57.407879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60604 (local address 127.3.243.67:42415)
0619 18:46:57.409956 (+ 2077us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:57.409964 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:57.410666 (+ 702us) server_negotiation.cc:407] Connection header received
0619 18:46:57.410918 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:57.410927 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:57.411197 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:57.411379 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:46:57.412357 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:57.414070 (+ 1713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:57.417046 (+ 2976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:57.418554 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:57.418655 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:57.418890 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:57.420737 (+ 1847us) server_negotiation.cc:299] Negotiation successful
0619 18:46:57.420920 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1891,"thread_start_us":1722,"threads_started":1}
I20260619 18:46:57.421178 5088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:57.407938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.66:42479 (local address 127.0.0.1:48462)
0619 18:46:57.411168 (+ 3230us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:57.411184 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 18:46:57.411370 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:46:57.412208 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:46:57.412213 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:46:57.412237 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:46:57.412641 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:57.412655 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:57.414678 (+ 2023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:57.414684 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:57.417331 (+ 2647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:57.417347 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:57.419450 (+ 2103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:57.419457 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:57.419628 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:57.419664 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 18:46:57.420683 (+ 1019us) client_negotiation.cc:240] Negotiation successful
0619 18:46:57.420825 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3114,"thread_start_us":2184,"threads_started":1}
I20260619 18:46:57.425338 5011 tablet_service.cc:1467] Processing CreateTablet for tablet 9b18dd4bbf3741c3850d8ec3e8a6dad2 (DEFAULT_TABLE table=test-table [id=739d5274c81348e0af4ff7b7b3a28361]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:46:57.425417 4878 tablet_service.cc:1467] Processing CreateTablet for tablet 9b18dd4bbf3741c3850d8ec3e8a6dad2 (DEFAULT_TABLE table=test-table [id=739d5274c81348e0af4ff7b7b3a28361]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:46:57.426641 4878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b18dd4bbf3741c3850d8ec3e8a6dad2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:46:57.426643 5011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b18dd4bbf3741c3850d8ec3e8a6dad2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:46:57.428699 5089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:57.412312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.65:32987 (local address 127.0.0.1:38916)
0619 18:46:57.421858 (+ 9546us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:57.421879 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 18:46:57.421985 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:46:57.422588 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:46:57.422592 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:46:57.422618 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:46:57.422765 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:57.422774 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:57.424547 (+ 1773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:57.424552 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:57.426267 (+ 1715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:57.426281 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:57.427977 (+ 1696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:57.427983 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:57.428178 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:57.428214 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 18:46:57.428307 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 18:46:57.428444 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9414,"thread_start_us":9519,"threads_started":1}
I20260619 18:46:57.428795 5092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:57.411516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38916 (local address 127.3.243.65:32987)
0619 18:46:57.421649 (+ 10133us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:57.421658 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:57.422038 (+ 380us) server_negotiation.cc:407] Connection header received
0619 18:46:57.422147 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:57.422155 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:57.422396 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:57.422578 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:46:57.422925 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:57.424386 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:57.426439 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:57.427807 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:57.427940 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:57.428125 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:57.428361 (+ 236us) server_negotiation.cc:299] Negotiation successful
0619 18:46:57.428520 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9953,"thread_start_us":9829,"threads_started":1}
I20260619 18:46:57.432293 4744 tablet_service.cc:1467] Processing CreateTablet for tablet 9b18dd4bbf3741c3850d8ec3e8a6dad2 (DEFAULT_TABLE table=test-table [id=739d5274c81348e0af4ff7b7b3a28361]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:46:57.433667 4744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b18dd4bbf3741c3850d8ec3e8a6dad2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:46:57.440709 5094 tablet_bootstrap.cc:492] T 9b18dd4bbf3741c3850d8ec3e8a6dad2 P 6c1a2c24714b47678fef1d20ed8bce0e: Bootstrap starting.
I20260619 18:46:57.443845 5094 tablet_bootstrap.cc:654] T 9b18dd4bbf3741c3850d8ec3e8a6dad2 P 6c1a2c24714b47678fef1d20ed8bce0e: Neither blocks nor log segments found. Creating new log.
I20260619 18:46:57.444859 5095 tablet_bootstrap.cc:492] T 9b18dd4bbf3741c3850d8ec3e8a6dad2 P 326b688a705a47a6b34539130d5540b0: Bootstrap starting.
I20260619 18:46:57.445068 5094 log.cc:824] T 9b18dd4bbf3741c3850d8ec3e8a6dad2 P 6c1a2c24714b47678fef1d20ed8bce0e: Log is configured to *not* fsync() on all Append() calls
I20260619 18:46:57.448408 5095 tablet_bootstrap.cc:654] T 9b18dd4bbf3741c3850d8ec3e8a6dad2 P 326b688a705a47a6b34539130d5540b0: Neither blocks nor log segments found. Creating new log.
I20260619 18:46:57.449730 5095 log.cc:824] T 9b18dd4bbf3741c3850d8ec3e8a6dad2 P 326b688a705a47a6b34539130d5540b0: Log is configured to *not* fsync() on all Append() calls
I20260619 18:46:57.453950 5097 tablet_bootstrap.cc:492] T 9b18dd4bbf3741c3850d8ec3e8a6dad2 P 316db71149464366baf681833b019e6a: Bootstrap starting.
I20260619 18:46:57.455132 5094 tablet_bootstrap.cc:492] T 9b18dd4bbf3741c3850d8ec3e8a6dad2 P 6c1a2c24714b47678fef1d20ed8bce0e: No bootstrap required, opened a new log
I20260619 18:46:57.455237 5095 tablet_bootstrap.cc:492] T 9b18dd4bbf3741c3850d8ec3e8a6dad2 P 326b688a705a47a6b34539130d5540b0: No bootstrap required, opened a new log
I20260619 18:46:57.455403 5094 ts_tablet_manager.cc:1397] T 9b18dd4bbf3741c3850d8ec3e8a6dad2 P 6c1a2c24714b47678fef1d20ed8bce0e: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.008s
I20260619 18:46:57.455482 5095 ts_tablet_manager.cc:1397] T 9b18dd4bbf3741c3850d8ec3e8a6dad2 P 326b688a705a47a6b34539130d5540b0: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.003s
I20260619 18:46:57.457327 5097 tablet_bootstrap.cc:654] T 9b18dd4bbf3741c3850d8ec3e8a6dad2 P 316db71149464366baf681833b019e6a: Neither blocks nor log segments found. Creating new log.
I20260619 18:46:57.458452 5097 log.cc:824] T 9b18dd4bbf3741c3850d8ec3e8a6dad2 P 316db71149464366baf681833b019e6a: Log is configured to *not* fsync() on all Append() calls
I20260619 18:46:57.462962 5095 raft_consensus.cc:348] T 9b18dd4bbf3741c3850d8ec3e8a6dad2 P 326b688a705a47a6b34539130d5540b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "326b688a705a47a6b34539130d5540b0" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 42415 } } peers { permanent_uuid: "316db71149464366baf681833b019e6a" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 32987 } } peers { permanent_uuid: "6c1a2c24714b47678fef1d20ed8bce0e" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 42479 } }
I20260619 18:46:57.463184 5097 tablet_bootstrap.cc:492] T 9b18dd4bbf3741c3850d8ec3e8a6dad2 P 316db71149464366baf681833b019e6a: No bootstrap required, opened a new log
I20260619 18:46:57.463296 5095 raft_consensus.cc:374] T 9b18dd4bbf3741c3850d8ec3e8a6dad2 P 326b688a705a47a6b34539130d5540b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:46:57.463424 5097 ts_tablet_manager.cc:1397] T 9b18dd4bbf3741c3850d8ec3e8a6dad2 P 316db71149464366baf681833b019e6a: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260619 18:46:57.463416 5095 raft_consensus.cc:729] T 9b18dd4bbf3741c3850d8ec3e8a6dad2 P 326b688a705a47a6b34539130d5540b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 326b688a705a47a6b34539130d5540b0, State: Initialized, Role: FOLLOWER
I20260619 18:46:57.463972 5095 consensus_queue.cc:260] T 9b18dd4bbf3741c3850d8ec3e8a6dad2 P 326b688a705a47a6b34539130d5540b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "326b688a705a47a6b34539130d5540b0" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 42415 } } peers { permanent_uuid: "316db71149464366baf681833b019e6a" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 32987 } } peers { permanent_uuid: "6c1a2c24714b47678fef1d20ed8bce0e" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 42479 } }
I20260619 18:46:57.465044 5094 raft_consensus.cc:348] T 9b18dd4bbf3741c3850d8ec3e8a6dad2 P 6c1a2c24714b47678fef1d20ed8bce0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "326b688a705a47a6b34539130d5540b0" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 42415 } } peers { permanent_uuid: "316db71149464366baf681833b019e6a" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 32987 } } peers { permanent_uuid: "6c1a2c24714b47678fef1d20ed8bce0e" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 42479 } }
I20260619 18:46:57.465457 5094 raft_consensus.cc:374] T 9b18dd4bbf3741c3850d8ec3e8a6dad2 P 6c1a2c24714b47678fef1d20ed8bce0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:46:57.465617 5094 raft_consensus.cc:729] T 9b18dd4bbf3741c3850d8ec3e8a6dad2 P 6c1a2c24714b47678fef1d20ed8bce0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c1a2c24714b47678fef1d20ed8bce0e, State: Initialized, Role: FOLLOWER
I20260619 18:46:57.466146 5076 heartbeater.cc:503] Master 127.3.243.126:37415 was elected leader, sending a full tablet report...
I20260619 18:46:57.466336 5094 consensus_queue.cc:260] T 9b18dd4bbf3741c3850d8ec3e8a6dad2 P 6c1a2c24714b47678fef1d20ed8bce0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "326b688a705a47a6b34539130d5540b0" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 42415 } } peers { permanent_uuid: "316db71149464366baf681833b019e6a" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 32987 } } peers { permanent_uuid: "6c1a2c24714b47678fef1d20ed8bce0e" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 42479 } }
I20260619 18:46:57.466602 4809 heartbeater.cc:503] Master 127.3.243.126:37415 was elected leader, sending a full tablet report...
I20260619 18:46:57.467828 5095 ts_tablet_manager.cc:1428] T 9b18dd4bbf3741c3850d8ec3e8a6dad2 P 326b688a705a47a6b34539130d5540b0: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260619 18:46:57.468783 4943 heartbeater.cc:503] Master 127.3.243.126:37415 was elected leader, sending a full tablet report...
I20260619 18:46:57.469457 5094 ts_tablet_manager.cc:1428] T 9b18dd4bbf3741c3850d8ec3e8a6dad2 P 6c1a2c24714b47678fef1d20ed8bce0e: Time spent starting tablet: real 0.014s user 0.012s sys 0.003s
I20260619 18:46:57.474108 5097 raft_consensus.cc:348] T 9b18dd4bbf3741c3850d8ec3e8a6dad2 P 316db71149464366baf681833b019e6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "326b688a705a47a6b34539130d5540b0" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 42415 } } peers { permanent_uuid: "316db71149464366baf681833b019e6a" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 32987 } } peers { permanent_uuid: "6c1a2c24714b47678fef1d20ed8bce0e" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 42479 } }
I20260619 18:46:57.474460 5097 raft_consensus.cc:374] T 9b18dd4bbf3741c3850d8ec3e8a6dad2 P 316db71149464366baf681833b019e6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:46:57.474556 5097 raft_consensus.cc:729] T 9b18dd4bbf3741c3850d8ec3e8a6dad2 P 316db71149464366baf681833b019e6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 316db71149464366baf681833b019e6a, State: Initialized, Role: FOLLOWER
I20260619 18:46:57.475037 5097 consensus_queue.cc:260] T 9b18dd4bbf3741c3850d8ec3e8a6dad2 P 316db71149464366baf681833b019e6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "326b688a705a47a6b34539130d5540b0" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 42415 } } peers { permanent_uuid: "316db71149464366baf681833b019e6a" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 32987 } } peers { permanent_uuid: "6c1a2c24714b47678fef1d20ed8bce0e" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 42479 } }
I20260619 18:46:57.476608 5097 ts_tablet_manager.cc:1428] T 9b18dd4bbf3741c3850d8ec3e8a6dad2 P 316db71149464366baf681833b019e6a: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
W20260619 18:46:57.593765 5077 tablet.cc:2369] T 9b18dd4bbf3741c3850d8ec3e8a6dad2 P 326b688a705a47a6b34539130d5540b0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:46:57.622838 5100 raft_consensus.cc:482] T 9b18dd4bbf3741c3850d8ec3e8a6dad2 P 326b688a705a47a6b34539130d5540b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:46:57.623155 5100 raft_consensus.cc:504] T 9b18dd4bbf3741c3850d8ec3e8a6dad2 P 326b688a705a47a6b34539130d5540b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "326b688a705a47a6b34539130d5540b0" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 42415 } } peers { permanent_uuid: "316db71149464366baf681833b019e6a" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 32987 } } peers { permanent_uuid: "6c1a2c24714b47678fef1d20ed8bce0e" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 42479 } }
I20260619 18:46:57.624946 5100 leader_election.cc:288] T 9b18dd4bbf3741c3850d8ec3e8a6dad2 P 326b688a705a47a6b34539130d5540b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 316db71149464366baf681833b019e6a (127.3.243.65:32987), 6c1a2c24714b47678fef1d20ed8bce0e (127.3.243.66:42479)
I20260619 18:46:57.631350 5092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:57.625390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.67:39803 (local address 127.3.243.65:32987)
0619 18:46:57.625589 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:57.625595 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:57.625676 (+ 81us) server_negotiation.cc:407] Connection header received
0619 18:46:57.625886 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:57.625892 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:57.625985 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:57.626098 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:46:57.626481 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:57.627407 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:57.629253 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:57.630521 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:57.630640 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:57.630693 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:57.631050 (+ 357us) server_negotiation.cc:299] Negotiation successful
0619 18:46:57.631143 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 18:46:57.631443 5078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:57.625233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.65:32987 (local address 127.3.243.67:39803)
0619 18:46:57.625561 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:57.625580 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 18:46:57.625741 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:46:57.626112 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:46:57.626115 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:46:57.626147 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:46:57.626337 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:57.626347 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:57.627525 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:57.627531 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:57.629104 (+ 1573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:57.629115 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:57.630686 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:57.630692 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:57.630896 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:57.630932 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 18:46:57.631033 (+ 101us) client_negotiation.cc:240] Negotiation successful
0619 18:46:57.631163 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187}
I20260619 18:46:57.632071 5091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:57.625390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.67:54963 (local address 127.3.243.66:42479)
0619 18:46:57.625589 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:57.625595 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:57.626181 (+ 586us) server_negotiation.cc:407] Connection header received
0619 18:46:57.626348 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:57.626353 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:57.626465 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:57.626533 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:46:57.627135 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:57.628068 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:57.629944 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:57.631217 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:57.631369 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:57.631434 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:57.631777 (+ 343us) server_negotiation.cc:299] Negotiation successful
0619 18:46:57.631875 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 18:46:57.632134 5103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:57.625322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.66:42479 (local address 127.3.243.67:54963)
0619 18:46:57.626075 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:57.626093 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 18:46:57.626221 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:46:57.626572 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:46:57.626576 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:46:57.626600 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:46:57.626963 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:57.626975 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:57.628240 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:57.628244 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:57.629799 (+ 1555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:46:57.629814 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:46:57.631434 (+ 1620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:46:57.631440 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:46:57.631615 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:57.631646 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 18:46:57.631738 (+ 92us) client_negotiation.cc:240] Negotiation successful
0619 18:46:57.631854 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"thread_start_us":423,"threads_started":1}
I20260619 18:46:57.632593 4764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b18dd4bbf3741c3850d8ec3e8a6dad2" candidate_uuid: "326b688a705a47a6b34539130d5540b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316db71149464366baf681833b019e6a" is_pre_election: true
I20260619 18:46:57.633080 4898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b18dd4bbf3741c3850d8ec3e8a6dad2" candidate_uuid: "326b688a705a47a6b34539130d5540b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c1a2c24714b47678fef1d20ed8bce0e" is_pre_election: true
I20260619 18:46:57.633229 4764 raft_consensus.cc:2454] T 9b18dd4bbf3741c3850d8ec3e8a6dad2 P 316db71149464366baf681833b019e6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 326b688a705a47a6b34539130d5540b0 in term 0.
I20260619 18:46:57.633595 4898 raft_consensus.cc:2454] T 9b18dd4bbf3741c3850d8ec3e8a6dad2 P 6c1a2c24714b47678fef1d20ed8bce0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 326b688a705a47a6b34539130d5540b0 in term 0.
W20260619 18:46:57.633983 4944 tablet.cc:2369] T 9b18dd4bbf3741c3850d8ec3e8a6dad2 P 6c1a2c24714b47678fef1d20ed8bce0e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:46:57.633986 4966 leader_election.cc:302] T 9b18dd4bbf3741c3850d8ec3e8a6dad2 P 326b688a705a47a6b34539130d5540b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316db71149464366baf681833b019e6a, 326b688a705a47a6b34539130d5540b0; no voters:
I20260619 18:46:57.634408 5100 raft_consensus.cc:2781] T 9b18dd4bbf3741c3850d8ec3e8a6dad2 P 326b688a705a47a6b34539130d5540b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:46:57.634547 5100 raft_consensus.cc:482] T 9b18dd4bbf3741c3850d8ec3e8a6dad2 P 326b688a705a47a6b34539130d5540b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:46:57.634637 5100 raft_consensus.cc:3037] T 9b18dd4bbf3741c3850d8ec3e8a6dad2 P 326b688a705a47a6b34539130d5540b0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:46:57.636834 5100 raft_consensus.cc:504] T 9b18dd4bbf3741c3850d8ec3e8a6dad2 P 326b688a705a47a6b34539130d5540b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "326b688a705a47a6b34539130d5540b0" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 42415 } } peers { permanent_uuid: "316db71149464366baf681833b019e6a" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 32987 } } peers { permanent_uuid: "6c1a2c24714b47678fef1d20ed8bce0e" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 42479 } }
I20260619 18:46:57.637477 5100 leader_election.cc:288] T 9b18dd4bbf3741c3850d8ec3e8a6dad2 P 326b688a705a47a6b34539130d5540b0 [CANDIDATE]: Term 1 election: Requested vote from peers 316db71149464366baf681833b019e6a (127.3.243.65:32987), 6c1a2c24714b47678fef1d20ed8bce0e (127.3.243.66:42479)
I20260619 18:46:57.637813 4764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b18dd4bbf3741c3850d8ec3e8a6dad2" candidate_uuid: "326b688a705a47a6b34539130d5540b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316db71149464366baf681833b019e6a"
I20260619 18:46:57.637876 4898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b18dd4bbf3741c3850d8ec3e8a6dad2" candidate_uuid: "326b688a705a47a6b34539130d5540b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c1a2c24714b47678fef1d20ed8bce0e"
I20260619 18:46:57.637997 4764 raft_consensus.cc:3037] T 9b18dd4bbf3741c3850d8ec3e8a6dad2 P 316db71149464366baf681833b019e6a [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:46:57.638049 4898 raft_consensus.cc:3037] T 9b18dd4bbf3741c3850d8ec3e8a6dad2 P 6c1a2c24714b47678fef1d20ed8bce0e [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:46:57.640223 4764 raft_consensus.cc:2454] T 9b18dd4bbf3741c3850d8ec3e8a6dad2 P 316db71149464366baf681833b019e6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 326b688a705a47a6b34539130d5540b0 in term 1.
I20260619 18:46:57.640347 4898 raft_consensus.cc:2454] T 9b18dd4bbf3741c3850d8ec3e8a6dad2 P 6c1a2c24714b47678fef1d20ed8bce0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 326b688a705a47a6b34539130d5540b0 in term 1.
I20260619 18:46:57.640604 4966 leader_election.cc:302] T 9b18dd4bbf3741c3850d8ec3e8a6dad2 P 326b688a705a47a6b34539130d5540b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316db71149464366baf681833b019e6a, 326b688a705a47a6b34539130d5540b0; no voters:
I20260619 18:46:57.640861 5100 raft_consensus.cc:2781] T 9b18dd4bbf3741c3850d8ec3e8a6dad2 P 326b688a705a47a6b34539130d5540b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:46:57.641273 5100 raft_consensus.cc:686] T 9b18dd4bbf3741c3850d8ec3e8a6dad2 P 326b688a705a47a6b34539130d5540b0 [term 1 LEADER]: Becoming Leader. State: Replica: 326b688a705a47a6b34539130d5540b0, State: Running, Role: LEADER
I20260619 18:46:57.641755 5100 consensus_queue.cc:237] T 9b18dd4bbf3741c3850d8ec3e8a6dad2 P 326b688a705a47a6b34539130d5540b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "326b688a705a47a6b34539130d5540b0" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 42415 } } peers { permanent_uuid: "316db71149464366baf681833b019e6a" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 32987 } } peers { permanent_uuid: "6c1a2c24714b47678fef1d20ed8bce0e" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 42479 } }
I20260619 18:46:57.645982 4620 catalog_manager.cc:5495] T 9b18dd4bbf3741c3850d8ec3e8a6dad2 P 326b688a705a47a6b34539130d5540b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 326b688a705a47a6b34539130d5540b0 (127.3.243.67). New cstate: current_term: 1 leader_uuid: "326b688a705a47a6b34539130d5540b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "326b688a705a47a6b34539130d5540b0" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 42415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "316db71149464366baf681833b019e6a" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 32987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c1a2c24714b47678fef1d20ed8bce0e" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 42479 } health_report { overall_health: UNKNOWN } } }
W20260619 18:46:57.688915 4810 tablet.cc:2369] T 9b18dd4bbf3741c3850d8ec3e8a6dad2 P 316db71149464366baf681833b019e6a: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 19 18:46:57 dist-test-slave-m9wg krb5kdc[4575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894815, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.65@KRBTEST.COM
I20260619 18:46:57.720954 5092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:57.705185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38918 (local address 127.3.243.65:32987)
0619 18:46:57.705459 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:57.705466 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:57.705928 (+ 462us) server_negotiation.cc:407] Connection header received
0619 18:46:57.706059 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:57.706064 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:57.706127 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:57.706209 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:57.707662 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:57.708448 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:57.709744 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:57.710163 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:57.710263 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:57.714010 (+ 3747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:57.714024 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:57.714040 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:46:57.714080 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:57.716927 (+ 2847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:57.717745 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:57.717754 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:57.717758 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:57.717953 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:57.718407 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:57.718412 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:57.718415 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:57.718942 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:57.719236 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:57.720344 (+ 1108us) server_negotiation.cc:299] Negotiation successful
0619 18:46:57.720574 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
Jun 19 18:46:57 dist-test-slave-m9wg krb5kdc[4575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894817, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 18:46:57 dist-test-slave-m9wg krb5kdc[4575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894817, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.243.65@KRBTEST.COM
I20260619 18:46:57.773620 5092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:57.757790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38930 (local address 127.3.243.65:32987)
0619 18:46:57.757998 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:57.758004 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:57.758231 (+ 227us) server_negotiation.cc:407] Connection header received
0619 18:46:57.758422 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:57.758426 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:57.758488 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:57.758571 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:57.760322 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:57.761116 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:57.762955 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:57.763353 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:57.763445 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:57.767928 (+ 4483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:57.767935 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:57.767941 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:46:57.767982 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:57.771158 (+ 3176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:57.771700 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:57.771705 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:57.771709 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:57.771872 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:57.772233 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:57.772238 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:57.772241 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:57.772589 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:57.772880 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:57.773025 (+ 145us) server_negotiation.cc:299] Negotiation successful
0619 18:46:57.773257 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
Jun 19 18:46:57 dist-test-slave-m9wg krb5kdc[4575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894817, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 18:46:57 dist-test-slave-m9wg krb5kdc[4575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894817, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.65@KRBTEST.COM
I20260619 18:46:57.819419 5092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:57.804494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38932 (local address 127.3.243.65:32987)
0619 18:46:57.804721 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:57.804727 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:57.804990 (+ 263us) server_negotiation.cc:407] Connection header received
0619 18:46:57.805189 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:57.805193 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:57.805254 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:57.805325 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:57.807013 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:57.807678 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:57.809462 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:57.809865 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:57.809970 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:57.814541 (+ 4571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:57.814546 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:57.814549 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:46:57.814590 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:57.816990 (+ 2400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:57.817759 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:57.817763 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:57.817765 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:57.817876 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:57.818234 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:57.818237 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:57.818239 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:57.818502 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:57.818683 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:57.818988 (+ 305us) server_negotiation.cc:299] Negotiation successful
0619 18:46:57.819157 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
W20260619 18:46:57.821141 4724 tablet_service.cc:2908] Rejecting scan request for tablet 9b18dd4bbf3741c3850d8ec3e8a6dad2: Uninitialized: clean time has not yet been initialized
I20260619 18:46:57.824378 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 4681
I20260619 18:46:57.841410 5092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:57.825257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38948 (local address 127.3.243.65:32987)
0619 18:46:57.825434 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:57.825439 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:57.829087 (+ 3648us) server_negotiation.cc:407] Connection header received
0619 18:46:57.829208 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:57.829214 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:57.829308 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:57.829417 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:57.830591 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:57.831670 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:57.832877 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:57.833512 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:57.833645 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:57.835597 (+ 1952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:57.835603 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:57.835609 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:46:57.835649 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:57.838789 (+ 3140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:57.839467 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:57.839472 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:57.839476 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:57.839617 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:57.839910 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:57.839914 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:57.839918 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:57.840351 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:57.840654 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:57.840783 (+ 129us) server_negotiation.cc:299] Negotiation successful
0619 18:46:57.841020 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 18:46:57.842082 4804 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:46:57.930044 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 4681
I20260619 18:46:57.952005 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 4815
Jun 19 18:46:57 dist-test-slave-m9wg krb5kdc[4575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894817, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.66@KRBTEST.COM
I20260619 18:46:57.967622 5091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:57.952866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48470 (local address 127.3.243.66:42479)
0619 18:46:57.953020 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:57.953025 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:57.953042 (+ 17us) server_negotiation.cc:407] Connection header received
0619 18:46:57.953258 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:57.953263 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:57.953327 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:57.953425 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:57.954903 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:57.955611 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:57.957563 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:57.957953 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:57.958043 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:57.962381 (+ 4338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:57.962394 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:57.962416 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:46:57.962452 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:57.965077 (+ 2625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:57.965715 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:57.965722 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:57.965725 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:57.965847 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:57.966184 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:57.966188 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:57.966190 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:57.966558 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:57.966816 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:57.967122 (+ 306us) server_negotiation.cc:299] Negotiation successful
0619 18:46:57.967325 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 18:46:57.968279 4938 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:46:58.054311 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 4815
I20260619 18:46:58.066893 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 4948
Jun 19 18:46:58 dist-test-slave-m9wg krb5kdc[4575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894817, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.67@KRBTEST.COM
I20260619 18:46:58.080861 5119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:58.067584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60612 (local address 127.3.243.67:42415)
0619 18:46:58.068046 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:58.068050 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:58.068065 (+ 15us) server_negotiation.cc:407] Connection header received
0619 18:46:58.068114 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:58.068118 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:58.068206 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:58.068297 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:58.069489 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:58.070151 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:58.071405 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:58.071806 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:58.071914 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:58.075765 (+ 3851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:58.075780 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:58.075796 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:46:58.075831 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:58.078280 (+ 2449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:58.078912 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:58.078920 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:58.078922 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:58.079058 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:58.079387 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:58.079390 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:58.079392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:58.079725 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:58.079997 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:58.080288 (+ 291us) server_negotiation.cc:299] Negotiation successful
0619 18:46:58.080499 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":216,"threads_started":1}
I20260619 18:46:58.081491 5071 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:46:58.170069 5121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:58.169366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.65:32987 (local address 127.3.243.67:48841)
0619 18:46:58.169788 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:58.169912 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.65:32987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":221,"threads_started":1}
W20260619 18:46:58.170336 4966 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.243.65:32987: connect: Connection refused (error 111)
I20260619 18:46:58.170499 5121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:58.169480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.66:42479 (local address 127.3.243.67:56555)
0619 18:46:58.170256 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:58.170353 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.66:42479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":698,"threads_started":1}
I20260619 18:46:58.171731 5121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:58.171388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.65:32987 (local address 127.3.243.67:39439)
0619 18:46:58.171545 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 18:46:58.171617 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.65:32987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260619 18:46:58.171897 4966 consensus_peers.cc:487] T 9b18dd4bbf3741c3850d8ec3e8a6dad2 P 326b688a705a47a6b34539130d5540b0 -> Peer 316db71149464366baf681833b019e6a (127.3.243.65:32987): Couldn't send request to peer 316db71149464366baf681833b019e6a. Status: Network error: Client connection negotiation failed: client connection to 127.3.243.65:32987: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260619 18:46:58.173242 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 4948
I20260619 18:46:58.185959 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 4588
I20260619 18:46:58.186960 4650 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:46:58.292711 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 4588
2026-06-19T18:46:58Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (3950 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894810715282-4045-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:46:58 dist-test-slave-m9wg krb5kdc[5127](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:46:58 dist-test-slave-m9wg krb5kdc[5127](info): set up 2 sockets
Jun 19 18:46:58 dist-test-slave-m9wg krb5kdc[5127](info): commencing operation
krb5kdc: starting...
W20260619 18:46:59.275954 4045 mini_kdc.cc:121] Time spent starting KDC: real 0.946s user 0.007s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894810715282-4045-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894810715282-4045-0/krb5kdc/test-user.keytab.
Jun 19 18:46:59 dist-test-slave-m9wg krb5kdc[5127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894819, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:46:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:46:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.243.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab.
Entry for principal kudu/127.3.243.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab.
I20260619 18:46:59.435443 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894810715282-4045-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894810715282-4045-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894810715282-4045-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.126:42779
--webserver_interface=127.3.243.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab
--principal=kudu/127.3.243.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:43067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.126:42779
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:46:59.742051 5140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:46:59.742338 5140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:46:59.742468 5140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:46:59.749601 5140 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:46:59.749706 5140 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 18:46:59.749770 5140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:46:59.749855 5140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:46:59.749917 5140 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:46:59.749998 5140 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:46:59.759742 5140 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:43067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894810715282-4045-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894810715282-4045-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.243.126:42779
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab
--principal=kudu/127.3.243.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.126:42779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.243.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894810715282-4045-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:46:59.760973 5140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:46:59.762307 5140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:46:59.773072 5145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:59.773139 5148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:46:59.773182 5146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:46:59.774194 5140 server_base.cc:1034] running on GCE node
I20260619 18:46:59.774909 5140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:46:59.776269 5140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:46:59.777516 5140 hybrid_clock.cc:648] HybridClock initialized: now 1781894819777443 us; error 38 us; skew 500 ppm
I20260619 18:46:59.777827 5140 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:46:59 dist-test-slave-m9wg krb5kdc[5127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894819, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:46:59.780901 5140 init.cc:377] Logged in from keytab as kudu/127.3.243.126@KRBTEST.COM (short username kudu)
I20260619 18:46:59.782246 5140 webserver.cc:460] Webserver started at http://127.3.243.126:39495/ using document root <none> and password file <none>
I20260619 18:46:59.782907 5140 fs_manager.cc:359] Metadata directory not provided
I20260619 18:46:59.783012 5140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:46:59.783283 5140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:46:59.786186 5140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894810715282-4045-0/minicluster-data/master-0/data/instance:
uuid: "9545518c740f4e57ab5c31e719e49d0c"
format_stamp: "Formatted at 2026-06-19 18:46:59 on dist-test-slave-m9wg"
server_key: "320a375a8098fedee58966df48b8c35a"
server_key_iv: "0c411b9926bda5cb1ff511eaa21c549f"
server_key_version: "encryptionkey@0"
I20260619 18:46:59.786945 5140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894810715282-4045-0/minicluster-data/master-0/wal/instance:
uuid: "9545518c740f4e57ab5c31e719e49d0c"
format_stamp: "Formatted at 2026-06-19 18:46:59 on dist-test-slave-m9wg"
server_key: "320a375a8098fedee58966df48b8c35a"
server_key_iv: "0c411b9926bda5cb1ff511eaa21c549f"
server_key_version: "encryptionkey@0"
I20260619 18:46:59.792078 5140 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.007s
I20260619 18:46:59.795279 5155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:46:59.796629 5140 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260619 18:46:59.796778 5140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894810715282-4045-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894810715282-4045-0/minicluster-data/master-0/wal
uuid: "9545518c740f4e57ab5c31e719e49d0c"
format_stamp: "Formatted at 2026-06-19 18:46:59 on dist-test-slave-m9wg"
server_key: "320a375a8098fedee58966df48b8c35a"
server_key_iv: "0c411b9926bda5cb1ff511eaa21c549f"
server_key_version: "encryptionkey@0"
I20260619 18:46:59.796970 5140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894810715282-4045-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894810715282-4045-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894810715282-4045-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:46:59.820998 5140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:46:59.825136 5140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:46:59.825378 5140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:46:59.844293 5140 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.126:42779
I20260619 18:46:59.844314 5216 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.126:42779 every 8 connection(s)
I20260619 18:46:59.845592 5140 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
I20260619 18:46:59.849110 5217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:46:59.850281 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 5140
I20260619 18:46:59.850464 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894810715282-4045-0/minicluster-data/master-0/wal/instance
I20260619 18:46:59.851056 4045 external_mini_cluster.cc:1428] Setting key 18201d70aab2d4f4cfa34cf56292e970
I20260619 18:46:59.861615 5217 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9545518c740f4e57ab5c31e719e49d0c: Bootstrap starting.
Jun 19 18:46:59 dist-test-slave-m9wg krb5kdc[5127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894819, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.126@KRBTEST.COM
I20260619 18:46:59.865226 5217 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9545518c740f4e57ab5c31e719e49d0c: Neither blocks nor log segments found. Creating new log.
I20260619 18:46:59.866169 5217 log.cc:824] T 00000000000000000000000000000000 P 9545518c740f4e57ab5c31e719e49d0c: Log is configured to *not* fsync() on all Append() calls
I20260619 18:46:59.868742 5217 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9545518c740f4e57ab5c31e719e49d0c: No bootstrap required, opened a new log
I20260619 18:46:59.872859 5220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:46:59.852869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41104 (local address 127.3.243.126:42779)
0619 18:46:59.853603 (+ 734us) server_negotiation.cc:206] Beginning negotiation
0619 18:46:59.853614 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 18:46:59.853656 (+ 42us) server_negotiation.cc:407] Connection header received
0619 18:46:59.855252 (+ 1596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:46:59.855273 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:46:59.855759 (+ 486us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:46:59.856075 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:46:59.857908 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:59.859282 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:59.860998 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:46:59.861673 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:46:59.861830 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:46:59.866172 (+ 4342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:46:59.866180 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:46:59.866194 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:46:59.866229 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:46:59.869093 (+ 2864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:59.869712 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:59.869721 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:59.869727 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:59.869876 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:46:59.870224 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:46:59.870228 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:46:59.870230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:46:59.870745 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:46:59.871031 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0619 18:46:59.871486 (+ 455us) server_negotiation.cc:299] Negotiation successful
0619 18:46:59.871860 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":332,"threads_started":1}
I20260619 18:46:59.876675 5217 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9545518c740f4e57ab5c31e719e49d0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9545518c740f4e57ab5c31e719e49d0c" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 42779 } }
I20260619 18:46:59.876997 5217 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9545518c740f4e57ab5c31e719e49d0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:46:59.877090 5217 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9545518c740f4e57ab5c31e719e49d0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9545518c740f4e57ab5c31e719e49d0c, State: Initialized, Role: FOLLOWER
I20260619 18:46:59.877569 5217 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9545518c740f4e57ab5c31e719e49d0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9545518c740f4e57ab5c31e719e49d0c" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 42779 } }
I20260619 18:46:59.877774 5217 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9545518c740f4e57ab5c31e719e49d0c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:46:59.877888 5217 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9545518c740f4e57ab5c31e719e49d0c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:46:59.878037 5217 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9545518c740f4e57ab5c31e719e49d0c [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:46:59.880101 5217 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9545518c740f4e57ab5c31e719e49d0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9545518c740f4e57ab5c31e719e49d0c" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 42779 } }
I20260619 18:46:59.880654 5217 leader_election.cc:302] T 00000000000000000000000000000000 P 9545518c740f4e57ab5c31e719e49d0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9545518c740f4e57ab5c31e719e49d0c; no voters:
I20260619 18:46:59.881245 5217 leader_election.cc:288] T 00000000000000000000000000000000 P 9545518c740f4e57ab5c31e719e49d0c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:46:59.881330 5222 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9545518c740f4e57ab5c31e719e49d0c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:46:59.881973 5222 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9545518c740f4e57ab5c31e719e49d0c [term 1 LEADER]: Becoming Leader. State: Replica: 9545518c740f4e57ab5c31e719e49d0c, State: Running, Role: LEADER
I20260619 18:46:59.882503 5222 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9545518c740f4e57ab5c31e719e49d0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9545518c740f4e57ab5c31e719e49d0c" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 42779 } }
I20260619 18:46:59.883100 5217 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9545518c740f4e57ab5c31e719e49d0c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:46:59.884969 5223 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9545518c740f4e57ab5c31e719e49d0c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9545518c740f4e57ab5c31e719e49d0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9545518c740f4e57ab5c31e719e49d0c" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 42779 } } }
I20260619 18:46:59.885344 5223 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9545518c740f4e57ab5c31e719e49d0c [sys.catalog]: This master's current role is: LEADER
I20260619 18:46:59.885339 5224 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9545518c740f4e57ab5c31e719e49d0c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9545518c740f4e57ab5c31e719e49d0c. Latest consensus state: current_term: 1 leader_uuid: "9545518c740f4e57ab5c31e719e49d0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9545518c740f4e57ab5c31e719e49d0c" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 42779 } } }
I20260619 18:46:59.885943 5224 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9545518c740f4e57ab5c31e719e49d0c [sys.catalog]: This master's current role is: LEADER
I20260619 18:46:59.886560 5231 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:46:59.891492 5231 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:46:59.900396 5231 catalog_manager.cc:1347] Generated new cluster ID: 61f9c2e2d65c451aa162568f96d42eec
I20260619 18:46:59.900565 5231 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:46:59.922849 5231 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:46:59.924767 5231 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:46:59.931648 5231 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9545518c740f4e57ab5c31e719e49d0c: Generated new TSK 0
I20260619 18:46:59.932708 5231 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.243.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.65.keytab.
Entry for principal kudu/127.3.243.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.65.keytab.
I20260619 18:46:59.972265 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894810715282-4045-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894810715282-4045-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894810715282-4045-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894810715282-4045-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.65:0
--local_ip_for_outbound_sockets=127.3.243.65
--webserver_interface=127.3.243.65
--webserver_port=0
--tserver_master_addrs=127.3.243.126:42779
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.65.keytab
--principal=kudu/127.3.243.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:43067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:47:00.273958 5243 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:47:00.274237 5243 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:47:00.274374 5243 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:47:00.281395 5243 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:47:00.281515 5243 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:47:00.281638 5243 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 18:47:00.281745 5243 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.65
I20260619 18:47:00.291361 5243 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:43067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894810715282-4045-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894810715282-4045-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.65.keytab
--principal=kudu/127.3.243.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894810715282-4045-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.243.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.243.126:42779
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.65
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894810715282-4045-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:47:00.292660 5243 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:47:00.293918 5243 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:47:00.306166 5251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:00.306134 5248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:00.306196 5249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:47:00.306813 5243 server_base.cc:1034] running on GCE node
I20260619 18:47:00.307348 5243 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:47:00.308187 5243 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:47:00.309392 5243 hybrid_clock.cc:648] HybridClock initialized: now 1781894820309316 us; error 57 us; skew 500 ppm
I20260619 18:47:00.309659 5243 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:47:00 dist-test-slave-m9wg krb5kdc[5127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894820, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:47:00.312567 5243 init.cc:377] Logged in from keytab as kudu/127.3.243.65@KRBTEST.COM (short username kudu)
I20260619 18:47:00.313894 5243 webserver.cc:460] Webserver started at http://127.3.243.65:36547/ using document root <none> and password file <none>
I20260619 18:47:00.314531 5243 fs_manager.cc:359] Metadata directory not provided
I20260619 18:47:00.314620 5243 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:47:00.314908 5243 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:47:00.317643 5243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894810715282-4045-0/minicluster-data/ts-0/data/instance:
uuid: "05fd30d5d3b84e93a6ec3a77870f6c34"
format_stamp: "Formatted at 2026-06-19 18:47:00 on dist-test-slave-m9wg"
server_key: "6defa00a96e3a628c6478b15031b33b8"
server_key_iv: "17f317ec2e511cf39fb0c38db0c42549"
server_key_version: "encryptionkey@0"
I20260619 18:47:00.318368 5243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894810715282-4045-0/minicluster-data/ts-0/wal/instance:
uuid: "05fd30d5d3b84e93a6ec3a77870f6c34"
format_stamp: "Formatted at 2026-06-19 18:47:00 on dist-test-slave-m9wg"
server_key: "6defa00a96e3a628c6478b15031b33b8"
server_key_iv: "17f317ec2e511cf39fb0c38db0c42549"
server_key_version: "encryptionkey@0"
I20260619 18:47:00.323086 5243 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260619 18:47:00.326236 5258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:00.327466 5243 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 18:47:00.327601 5243 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894810715282-4045-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894810715282-4045-0/minicluster-data/ts-0/wal
uuid: "05fd30d5d3b84e93a6ec3a77870f6c34"
format_stamp: "Formatted at 2026-06-19 18:47:00 on dist-test-slave-m9wg"
server_key: "6defa00a96e3a628c6478b15031b33b8"
server_key_iv: "17f317ec2e511cf39fb0c38db0c42549"
server_key_version: "encryptionkey@0"
I20260619 18:47:00.327775 5243 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894810715282-4045-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894810715282-4045-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894810715282-4045-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:47:00.341033 5243 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:47:00.344491 5243 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:47:00.344709 5243 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:47:00.347146 5243 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:47:00.347237 5243 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:00.347385 5243 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:47:00.347512 5243 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Jun 19 18:47:00 dist-test-slave-m9wg krb5kdc[5127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894820, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.65@KRBTEST.COM for kudu/127.3.243.126@KRBTEST.COM
I20260619 18:47:00.371417 5267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:00.348174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:42779 (local address 127.3.243.65:59143)
0619 18:47:00.348911 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:00.348950 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:00.350744 (+ 1794us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:00.351452 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:00.351460 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:00.352184 (+ 724us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:47:00.353018 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:00.353043 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:00.354305 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:00.354311 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:00.356661 (+ 2350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:00.356676 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:00.357912 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:00.357920 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:00.358207 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:00.358988 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:47:00.359021 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:47:00.362197 (+ 3176us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:47:00.366052 (+ 3855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:00.366058 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:00.366070 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:00.366664 (+ 594us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:00.367111 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:00.367118 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:00.367125 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:00.367485 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:00.368119 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:47:00.368127 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:47:00.368669 (+ 542us) client_negotiation.cc:769] Sending connection context
0619 18:47:00.369010 (+ 341us) client_negotiation.cc:240] Negotiation successful
0619 18:47:00.369627 (+ 617us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":322,"threads_started":1}
I20260619 18:47:00.372407 5220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:00.347554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.65:59143 (local address 127.3.243.126:42779)
0619 18:47:00.347718 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:00.347725 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:00.349076 (+ 1351us) server_negotiation.cc:407] Connection header received
0619 18:47:00.351021 (+ 1945us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:00.351027 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:00.351111 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:00.351198 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:00.353206 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:00.354138 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:00.356844 (+ 2706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:00.357436 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:00.357568 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:00.362472 (+ 4904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:00.362479 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:00.362483 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:00.362515 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:00.365885 (+ 3370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:00.366786 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:00.366792 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:00.366795 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:00.366985 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:00.367612 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:00.367615 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:00.367617 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:00.367963 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:00.368117 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:00.371812 (+ 3695us) server_negotiation.cc:299] Negotiation successful
0619 18:47:00.372053 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 18:47:00.387312 5243 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.65:39053
I20260619 18:47:00.387317 5377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.65:39053 every 8 connection(s)
I20260619 18:47:00.388878 5243 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894810715282-4045-0/minicluster-data/ts-0/data/info.pb
I20260619 18:47:00.391224 5378 heartbeater.cc:347] Connected to a master server at 127.3.243.126:42779
I20260619 18:47:00.391453 5378 heartbeater.cc:464] Registering TS with master...
I20260619 18:47:00.392201 5378 heartbeater.cc:511] Master 127.3.243.126:42779 requested a full tablet report, sending...
I20260619 18:47:00.394397 5172 ts_manager.cc:194] Registered new tserver with Master: 05fd30d5d3b84e93a6ec3a77870f6c34 (127.3.243.65:39053)
I20260619 18:47:00.396235 5172 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.65@KRBTEST.COM'} at 127.3.243.65:59143
I20260619 18:47:00.396700 5220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:00.382844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.65:59927 (local address 127.3.243.126:42779)
0619 18:47:00.383022 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:00.383027 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:00.383811 (+ 784us) server_negotiation.cc:407] Connection header received
0619 18:47:00.384010 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:00.384016 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:00.384094 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:00.384555 (+ 461us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:00.385775 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:00.386778 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:00.388651 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:00.389075 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:00.389163 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:00.391174 (+ 2011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:00.391180 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:00.391184 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:00.391220 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:00.393958 (+ 2738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:00.394675 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:00.394681 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:00.394685 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:00.394840 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:00.395251 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:00.395255 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:00.395257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:00.395588 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:00.395700 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:00.396122 (+ 422us) server_negotiation.cc:299] Negotiation successful
0619 18:47:00.396416 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 18:47:00.396744 5367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:00.382905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:42779 (local address 127.3.243.65:59927)
0619 18:47:00.383706 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:00.383745 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:00.383868 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:00.384619 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:00.384624 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:00.385268 (+ 644us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:47:00.385642 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:00.385654 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:00.386914 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:00.386919 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:00.388510 (+ 1591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:00.388524 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:00.389245 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:00.389252 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:00.389467 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:00.390123 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:47:00.390139 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:47:00.391022 (+ 883us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:47:00.394078 (+ 3056us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:00.394083 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:00.394086 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:00.394557 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:00.394953 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:00.394958 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:00.394961 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:00.395109 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:00.395699 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:47:00.395704 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:47:00.395889 (+ 185us) client_negotiation.cc:769] Sending connection context
0619 18:47:00.396032 (+ 143us) client_negotiation.cc:240] Negotiation successful
0619 18:47:00.396349 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":630,"thread_start_us":429,"threads_started":1}
I20260619 18:47:00.398206 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 5243
I20260619 18:47:00.398428 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894810715282-4045-0/minicluster-data/ts-0/wal/instance
I20260619 18:47:00.399199 4045 external_mini_cluster.cc:1428] Setting key 47c58a20bcc98c02ec6da13f29311992
WARNING: no policy specified for kudu/127.3.243.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.66.keytab.
Entry for principal kudu/127.3.243.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.66.keytab.
I20260619 18:47:00.433058 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894810715282-4045-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894810715282-4045-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894810715282-4045-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894810715282-4045-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.66:0
--local_ip_for_outbound_sockets=127.3.243.66
--webserver_interface=127.3.243.66
--webserver_port=0
--tserver_master_addrs=127.3.243.126:42779
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.66.keytab
--principal=kudu/127.3.243.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:43067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:47:00.732780 5382 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:47:00.733042 5382 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:47:00.733144 5382 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:47:00.740016 5382 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:47:00.740118 5382 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:47:00.740239 5382 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 18:47:00.740324 5382 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.66
I20260619 18:47:00.749533 5382 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:43067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894810715282-4045-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894810715282-4045-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.66.keytab
--principal=kudu/127.3.243.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894810715282-4045-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.243.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.243.126:42779
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.66
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894810715282-4045-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:47:00.750659 5382 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:47:00.751848 5382 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:47:00.763769 5388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:00.763823 5387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:00.763823 5390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:47:00.764595 5382 server_base.cc:1034] running on GCE node
I20260619 18:47:00.765115 5382 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:47:00.765918 5382 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:47:00.767138 5382 hybrid_clock.cc:648] HybridClock initialized: now 1781894820767075 us; error 40 us; skew 500 ppm
I20260619 18:47:00.767408 5382 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:47:00 dist-test-slave-m9wg krb5kdc[5127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894820, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:47:00.770308 5382 init.cc:377] Logged in from keytab as kudu/127.3.243.66@KRBTEST.COM (short username kudu)
I20260619 18:47:00.771551 5382 webserver.cc:460] Webserver started at http://127.3.243.66:41817/ using document root <none> and password file <none>
I20260619 18:47:00.772212 5382 fs_manager.cc:359] Metadata directory not provided
I20260619 18:47:00.772305 5382 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:47:00.772598 5382 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:47:00.775489 5382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894810715282-4045-0/minicluster-data/ts-1/data/instance:
uuid: "542019cd88ee453597eb659a94f7634b"
format_stamp: "Formatted at 2026-06-19 18:47:00 on dist-test-slave-m9wg"
server_key: "8d49098475d3efc55dbfff7ee910c03a"
server_key_iv: "34b5f45e97cf9c69052b22dce700f3f4"
server_key_version: "encryptionkey@0"
I20260619 18:47:00.776278 5382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894810715282-4045-0/minicluster-data/ts-1/wal/instance:
uuid: "542019cd88ee453597eb659a94f7634b"
format_stamp: "Formatted at 2026-06-19 18:47:00 on dist-test-slave-m9wg"
server_key: "8d49098475d3efc55dbfff7ee910c03a"
server_key_iv: "34b5f45e97cf9c69052b22dce700f3f4"
server_key_version: "encryptionkey@0"
I20260619 18:47:00.780920 5382 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 18:47:00.784076 5397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:00.785321 5382 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 18:47:00.785462 5382 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894810715282-4045-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894810715282-4045-0/minicluster-data/ts-1/wal
uuid: "542019cd88ee453597eb659a94f7634b"
format_stamp: "Formatted at 2026-06-19 18:47:00 on dist-test-slave-m9wg"
server_key: "8d49098475d3efc55dbfff7ee910c03a"
server_key_iv: "34b5f45e97cf9c69052b22dce700f3f4"
server_key_version: "encryptionkey@0"
I20260619 18:47:00.785645 5382 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894810715282-4045-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894810715282-4045-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894810715282-4045-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:47:00.809382 5382 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:47:00.813326 5382 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:47:00.813561 5382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:47:00.815959 5382 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:47:00.816035 5382 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:00.816206 5382 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:47:00.816324 5382 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
Jun 19 18:47:00 dist-test-slave-m9wg krb5kdc[5127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894820, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.66@KRBTEST.COM for kudu/127.3.243.126@KRBTEST.COM
I20260619 18:47:00.838500 5220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:00.816758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.66:36983 (local address 127.3.243.126:42779)
0619 18:47:00.816949 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:00.816955 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:00.818257 (+ 1302us) server_negotiation.cc:407] Connection header received
0619 18:47:00.820979 (+ 2722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:00.820985 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:00.821075 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:00.821190 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:00.823633 (+ 2443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:00.824558 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:00.826794 (+ 2236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:00.827273 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:00.827385 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:00.831742 (+ 4357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:00.831749 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:00.831754 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:00.831795 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:00.834751 (+ 2956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:00.835682 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:00.835686 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:00.835688 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:00.835826 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:00.836457 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:00.836463 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:00.836467 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:00.836776 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:00.836914 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:00.837849 (+ 935us) server_negotiation.cc:299] Negotiation successful
0619 18:47:00.838114 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 18:47:00.839169 5406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:00.817247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:42779 (local address 127.3.243.66:36983)
0619 18:47:00.818063 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:00.818118 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:00.820644 (+ 2526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:00.821515 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:00.821527 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:00.822382 (+ 855us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:47:00.823346 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:00.823380 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:00.824759 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:00.824764 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:00.826650 (+ 1886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:00.826670 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:00.827500 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:00.827508 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:00.827699 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:00.828460 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:47:00.828494 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:47:00.831517 (+ 3023us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:47:00.834893 (+ 3376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:00.834902 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:00.834927 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:00.835526 (+ 599us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:00.835966 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:00.835971 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:00.835975 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:00.836317 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:00.836988 (+ 671us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:47:00.837000 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:47:00.837463 (+ 463us) client_negotiation.cc:769] Sending connection context
0619 18:47:00.837682 (+ 219us) client_negotiation.cc:240] Negotiation successful
0619 18:47:00.838058 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":313,"threads_started":1}
I20260619 18:47:00.860853 5220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:00.844402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.66:51103 (local address 127.3.243.126:42779)
0619 18:47:00.844590 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:00.844595 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:00.845663 (+ 1068us) server_negotiation.cc:407] Connection header received
0619 18:47:00.845811 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:00.845817 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:00.845921 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:00.846019 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:00.847115 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:00.848353 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:00.850444 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:00.851114 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:00.851225 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:00.853829 (+ 2604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:00.853836 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:00.853842 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:00.853881 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:00.857442 (+ 3561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:00.858331 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:00.858336 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:00.858339 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:00.858513 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:00.858958 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:00.858963 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:00.858967 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:00.859355 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:00.859506 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:00.860376 (+ 870us) server_negotiation.cc:299] Negotiation successful
0619 18:47:00.860600 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 18:47:00.860852 5469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:00.844912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:42779 (local address 127.3.243.66:51103)
0619 18:47:00.845552 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:00.845571 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:00.845700 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:00.846043 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:00.846048 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:00.846589 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:47:00.846963 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:00.846979 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:00.848558 (+ 1579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:00.848564 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:00.850255 (+ 1691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:00.850271 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:00.851492 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:00.851499 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:00.851690 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:00.852405 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:47:00.852427 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:47:00.853611 (+ 1184us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:47:00.857604 (+ 3993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:00.857609 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:00.857614 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:00.858185 (+ 571us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:00.858643 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:00.858649 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:00.858653 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:00.858814 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:00.859567 (+ 753us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:47:00.859572 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:47:00.859800 (+ 228us) client_negotiation.cc:769] Sending connection context
0619 18:47:00.859957 (+ 157us) client_negotiation.cc:240] Negotiation successful
0619 18:47:00.860343 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":329,"threads_started":1}
I20260619 18:47:00.868912 5382 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.66:36729
I20260619 18:47:00.868932 5516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.66:36729 every 8 connection(s)
I20260619 18:47:00.870030 5382 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894810715282-4045-0/minicluster-data/ts-1/data/info.pb
I20260619 18:47:00.871686 5517 heartbeater.cc:347] Connected to a master server at 127.3.243.126:42779
I20260619 18:47:00.871861 5517 heartbeater.cc:464] Registering TS with master...
I20260619 18:47:00.872447 5517 heartbeater.cc:511] Master 127.3.243.126:42779 requested a full tablet report, sending...
I20260619 18:47:00.873565 5172 ts_manager.cc:194] Registered new tserver with Master: 542019cd88ee453597eb659a94f7634b (127.3.243.66:36729)
I20260619 18:47:00.874292 5172 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.66@KRBTEST.COM'} at 127.3.243.66:36983
I20260619 18:47:00.878116 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 5382
I20260619 18:47:00.878285 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894810715282-4045-0/minicluster-data/ts-1/wal/instance
I20260619 18:47:00.878719 4045 external_mini_cluster.cc:1428] Setting key a76323ae5ff9c5ef7795d554c33aea10
WARNING: no policy specified for kudu/127.3.243.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.67.keytab.
Entry for principal kudu/127.3.243.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.67.keytab.
I20260619 18:47:00.912771 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894810715282-4045-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894810715282-4045-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894810715282-4045-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894810715282-4045-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.67:0
--local_ip_for_outbound_sockets=127.3.243.67
--webserver_interface=127.3.243.67
--webserver_port=0
--tserver_master_addrs=127.3.243.126:42779
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.67.keytab
--principal=kudu/127.3.243.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:43067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:47:01.228981 5521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:47:01.229257 5521 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:47:01.229395 5521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:47:01.236470 5521 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:47:01.236579 5521 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:47:01.236696 5521 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 18:47:01.236804 5521 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.67
I20260619 18:47:01.246270 5521 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:43067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894810715282-4045-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894810715282-4045-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.67.keytab
--principal=kudu/127.3.243.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894810715282-4045-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.243.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.243.126:42779
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.67
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894810715282-4045-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:47:01.247480 5521 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:47:01.248803 5521 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:47:01.261058 5526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:01.261114 5527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:01.261023 5529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:47:01.265460 5521 server_base.cc:1034] running on GCE node
I20260619 18:47:01.266001 5521 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:47:01.266837 5521 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:47:01.268033 5521 hybrid_clock.cc:648] HybridClock initialized: now 1781894821267963 us; error 47 us; skew 500 ppm
I20260619 18:47:01.268325 5521 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:47:01 dist-test-slave-m9wg krb5kdc[5127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894821, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:47:01.271168 5521 init.cc:377] Logged in from keytab as kudu/127.3.243.67@KRBTEST.COM (short username kudu)
I20260619 18:47:01.272542 5521 webserver.cc:460] Webserver started at http://127.3.243.67:41009/ using document root <none> and password file <none>
I20260619 18:47:01.273212 5521 fs_manager.cc:359] Metadata directory not provided
I20260619 18:47:01.273300 5521 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:47:01.273581 5521 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:47:01.276475 5521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894810715282-4045-0/minicluster-data/ts-2/data/instance:
uuid: "afeb1cbb5dbb4580ba7cf5183c0ae346"
format_stamp: "Formatted at 2026-06-19 18:47:01 on dist-test-slave-m9wg"
server_key: "88c6a65a777e1a0241861919ba76285b"
server_key_iv: "65ad812680678bdc145b75a40de98ce5"
server_key_version: "encryptionkey@0"
I20260619 18:47:01.277189 5521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894810715282-4045-0/minicluster-data/ts-2/wal/instance:
uuid: "afeb1cbb5dbb4580ba7cf5183c0ae346"
format_stamp: "Formatted at 2026-06-19 18:47:01 on dist-test-slave-m9wg"
server_key: "88c6a65a777e1a0241861919ba76285b"
server_key_iv: "65ad812680678bdc145b75a40de98ce5"
server_key_version: "encryptionkey@0"
I20260619 18:47:01.282006 5521 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 18:47:01.285481 5536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:01.286789 5521 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 18:47:01.286916 5521 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894810715282-4045-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894810715282-4045-0/minicluster-data/ts-2/wal
uuid: "afeb1cbb5dbb4580ba7cf5183c0ae346"
format_stamp: "Formatted at 2026-06-19 18:47:01 on dist-test-slave-m9wg"
server_key: "88c6a65a777e1a0241861919ba76285b"
server_key_iv: "65ad812680678bdc145b75a40de98ce5"
server_key_version: "encryptionkey@0"
I20260619 18:47:01.287093 5521 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894810715282-4045-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894810715282-4045-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894810715282-4045-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:47:01.318096 5521 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:47:01.322314 5521 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:47:01.322542 5521 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:47:01.325160 5521 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:47:01.325235 5521 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:01.325357 5521 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:47:01.325408 5521 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 18:47:01 dist-test-slave-m9wg krb5kdc[5127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894821, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.67@KRBTEST.COM for kudu/127.3.243.126@KRBTEST.COM
I20260619 18:47:01.349831 5220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:01.325735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.67:60631 (local address 127.3.243.126:42779)
0619 18:47:01.325952 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:01.325957 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:01.327352 (+ 1395us) server_negotiation.cc:407] Connection header received
0619 18:47:01.329712 (+ 2360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:01.329718 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:01.329805 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:01.329925 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:01.332021 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:01.333013 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:01.335300 (+ 2287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:01.335898 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:01.336034 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:01.341105 (+ 5071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:01.341112 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:01.341117 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:01.341154 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:01.344366 (+ 3212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:01.345348 (+ 982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:01.345353 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:01.345356 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:01.345504 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:01.346108 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:01.346113 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:01.346117 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:01.346485 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:01.346643 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:01.349259 (+ 2616us) server_negotiation.cc:299] Negotiation successful
0619 18:47:01.349500 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260619 18:47:01.350333 5545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:01.326320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:42779 (local address 127.3.243.67:60631)
0619 18:47:01.327163 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:01.327215 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:01.329379 (+ 2164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:01.330206 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:01.330214 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:01.330969 (+ 755us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:47:01.331819 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:01.331852 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:01.333216 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:01.333222 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:01.335129 (+ 1907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:01.335150 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:01.336203 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:01.336212 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:01.336607 (+ 395us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:01.337543 (+ 936us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:47:01.337579 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:47:01.340861 (+ 3282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:47:01.344502 (+ 3641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:01.344510 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:01.344536 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:01.345175 (+ 639us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:01.345616 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:01.345622 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:01.345627 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:01.345941 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:01.346730 (+ 789us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:47:01.346738 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:47:01.347341 (+ 603us) client_negotiation.cc:769] Sending connection context
0619 18:47:01.347711 (+ 370us) client_negotiation.cc:240] Negotiation successful
0619 18:47:01.348411 (+ 700us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":342,"threads_started":1}
I20260619 18:47:01.363037 5521 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.67:44041
I20260619 18:47:01.363090 5655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.67:44041 every 8 connection(s)
I20260619 18:47:01.364427 5521 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894810715282-4045-0/minicluster-data/ts-2/data/info.pb
I20260619 18:47:01.366693 5656 heartbeater.cc:347] Connected to a master server at 127.3.243.126:42779
I20260619 18:47:01.366930 5656 heartbeater.cc:464] Registering TS with master...
I20260619 18:47:01.367199 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 5521
I20260619 18:47:01.367391 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894810715282-4045-0/minicluster-data/ts-2/wal/instance
I20260619 18:47:01.367748 5656 heartbeater.cc:511] Master 127.3.243.126:42779 requested a full tablet report, sending...
I20260619 18:47:01.367939 4045 external_mini_cluster.cc:1428] Setting key a2ec8c705d5430286bac3333905c0271
I20260619 18:47:01.369352 5171 ts_manager.cc:194] Registered new tserver with Master: afeb1cbb5dbb4580ba7cf5183c0ae346 (127.3.243.67:44041)
I20260619 18:47:01.370249 5171 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.67@KRBTEST.COM'} at 127.3.243.67:60631
I20260619 18:47:01.370707 4045 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 18:47:01.375243 5649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:01.357557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:42779 (local address 127.3.243.67:45381)
0619 18:47:01.361016 (+ 3459us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:01.361041 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:01.361426 (+ 385us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:01.361742 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:01.361747 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:01.362392 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:47:01.362808 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:01.362827 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:01.363934 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:01.363943 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:01.365477 (+ 1534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:01.365488 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:01.366486 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:01.366492 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:01.366657 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:01.367252 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:47:01.367269 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:47:01.368093 (+ 824us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:47:01.373092 (+ 4999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:01.373095 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:01.373098 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:01.373431 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:01.373832 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:01.373835 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:01.373837 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:01.373947 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:01.374479 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:47:01.374483 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:47:01.374625 (+ 142us) client_negotiation.cc:769] Sending connection context
0619 18:47:01.374730 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 18:47:01.374925 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3273,"thread_start_us":3051,"threads_started":1}
I20260619 18:47:01.375413 5220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:01.357581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.67:45381 (local address 127.3.243.126:42779)
0619 18:47:01.357770 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:01.357777 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:01.361139 (+ 3362us) server_negotiation.cc:407] Connection header received
0619 18:47:01.361546 (+ 407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:01.361550 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:01.361609 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:01.361685 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:01.362980 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:01.363800 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:01.365638 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:01.366156 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:01.366253 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:01.368264 (+ 2011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:01.368270 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:01.368275 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:01.368309 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:01.372984 (+ 4675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:01.373553 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:01.373558 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:01.373562 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:01.373738 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:01.374037 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:01.374042 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:01.374044 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:01.374365 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:01.374523 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:01.374860 (+ 337us) server_negotiation.cc:299] Negotiation successful
0619 18:47:01.375060 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
Jun 19 18:47:01 dist-test-slave-m9wg krb5kdc[5127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894821, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
I20260619 18:47:01.399467 5378 heartbeater.cc:503] Master 127.3.243.126:42779 was elected leader, sending a full tablet report...
Jun 19 18:47:01 dist-test-slave-m9wg krb5kdc[5127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894821, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.243.126@KRBTEST.COM
I20260619 18:47:01.413170 5220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:01.400935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41108 (local address 127.3.243.126:42779)
0619 18:47:01.401109 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:01.401115 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:01.401474 (+ 359us) server_negotiation.cc:407] Connection header received
0619 18:47:01.401614 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:01.401617 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:01.401680 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:01.401759 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:01.402938 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:01.403642 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:01.404828 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:01.405222 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:01.405307 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:01.408695 (+ 3388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:01.408700 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:01.408703 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:01.408735 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:01.410848 (+ 2113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:01.411411 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:01.411415 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:01.411417 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:01.411538 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:01.411873 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:01.411878 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:01.411881 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:01.412130 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:01.412390 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:01.412690 (+ 300us) server_negotiation.cc:299] Negotiation successful
0619 18:47:01.412895 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 18:47:01.416596 5171 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41108:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:47:01.423113 5171 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:47:01.459767 5670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:01.434083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46702 (local address 127.3.243.65:39053)
0619 18:47:01.451615 (+ 17532us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:01.451621 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:01.451643 (+ 22us) server_negotiation.cc:407] Connection header received
0619 18:47:01.451721 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:01.451727 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:01.451914 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:01.452119 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:47:01.452803 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:01.454321 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:01.457180 (+ 2859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:01.458592 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:01.458714 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:01.458925 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:01.459314 (+ 389us) server_negotiation.cc:299] Negotiation successful
0619 18:47:01.459479 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17409,"thread_start_us":18268,"threads_started":1}
I20260619 18:47:01.459823 5669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:01.433894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.65:39053 (local address 127.0.0.1:46702)
0619 18:47:01.449405 (+ 15511us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:01.449463 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:01.449661 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:01.452143 (+ 2482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:01.452177 (+ 34us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:01.452206 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:47:01.452605 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:01.452619 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:01.454451 (+ 1832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:01.454456 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:01.456994 (+ 2538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:01.457008 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:01.458792 (+ 1784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:01.458798 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:01.459065 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:01.459121 (+ 56us) client_negotiation.cc:769] Sending connection context
0619 18:47:01.459243 (+ 122us) client_negotiation.cc:240] Negotiation successful
0619 18:47:01.459421 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15208,"mutex_wait_us":23,"thread_start_us":15949,"threads_started":1}
I20260619 18:47:01.460958 5674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:01.434802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51218 (local address 127.3.243.66:36729)
0619 18:47:01.452900 (+ 18098us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:01.452907 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:01.453946 (+ 1039us) server_negotiation.cc:407] Connection header received
0619 18:47:01.454044 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:01.454052 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:01.454301 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:01.454466 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:47:01.455369 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:01.456531 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:01.458568 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:01.459790 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:01.459917 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:01.460110 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:01.460547 (+ 437us) server_negotiation.cc:299] Negotiation successful
0619 18:47:01.460698 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17943,"thread_start_us":17835,"threads_started":1}
I20260619 18:47:01.461207 5671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:01.434361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.66:36729 (local address 127.0.0.1:51218)
0619 18:47:01.453706 (+ 19345us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:01.453723 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:01.453833 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:01.454544 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:01.454548 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:01.454575 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:47:01.455143 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:01.455159 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:01.456647 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:01.456652 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:01.458431 (+ 1779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:01.458444 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:01.460261 (+ 1817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:01.460268 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:01.460373 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:01.460395 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 18:47:01.460479 (+ 84us) client_negotiation.cc:240] Negotiation successful
0619 18:47:01.460593 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19203,"mutex_wait_us":29,"thread_start_us":18697,"threads_started":1}
I20260619 18:47:01.462241 5673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:01.434697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42102 (local address 127.3.243.67:44041)
0619 18:47:01.444097 (+ 9400us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:01.444107 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:01.453831 (+ 9724us) server_negotiation.cc:407] Connection header received
0619 18:47:01.454094 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:01.454102 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:01.454421 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:01.454569 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:47:01.455540 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:01.457155 (+ 1615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:01.459480 (+ 2325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:01.460831 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:01.460923 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:01.461112 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:01.461792 (+ 680us) server_negotiation.cc:299] Negotiation successful
0619 18:47:01.461941 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9206,"thread_start_us":9025,"threads_started":1}
I20260619 18:47:01.462951 5672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:01.434177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.67:44041 (local address 127.0.0.1:42102)
0619 18:47:01.453726 (+ 19549us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:01.453745 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:01.453925 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:01.454716 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:01.454720 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:01.454753 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:47:01.455279 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:01.455291 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:01.457324 (+ 2033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:01.457329 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:01.459306 (+ 1977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:01.459319 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:01.461309 (+ 1990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:01.461316 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:01.461596 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:01.461627 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 18:47:01.462488 (+ 861us) client_negotiation.cc:240] Negotiation successful
0619 18:47:01.462612 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19018,"mutex_wait_us":29,"thread_start_us":19259,"threads_started":1}
I20260619 18:47:01.464011 5308 tablet_service.cc:1467] Processing CreateTablet for tablet c8f28bcf864d41a8a94b8bc4dacc1cb1 (DEFAULT_TABLE table=test-table [id=2ceb3108526547bb8e338ed702657fbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:47:01.464442 5447 tablet_service.cc:1467] Processing CreateTablet for tablet c8f28bcf864d41a8a94b8bc4dacc1cb1 (DEFAULT_TABLE table=test-table [id=2ceb3108526547bb8e338ed702657fbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:47:01.465263 5308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8f28bcf864d41a8a94b8bc4dacc1cb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:47:01.465750 5447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8f28bcf864d41a8a94b8bc4dacc1cb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:47:01.466260 5586 tablet_service.cc:1467] Processing CreateTablet for tablet c8f28bcf864d41a8a94b8bc4dacc1cb1 (DEFAULT_TABLE table=test-table [id=2ceb3108526547bb8e338ed702657fbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:47:01.467542 5586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8f28bcf864d41a8a94b8bc4dacc1cb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:47:01.475720 5675 tablet_bootstrap.cc:492] T c8f28bcf864d41a8a94b8bc4dacc1cb1 P 05fd30d5d3b84e93a6ec3a77870f6c34: Bootstrap starting.
I20260619 18:47:01.478720 5675 tablet_bootstrap.cc:654] T c8f28bcf864d41a8a94b8bc4dacc1cb1 P 05fd30d5d3b84e93a6ec3a77870f6c34: Neither blocks nor log segments found. Creating new log.
I20260619 18:47:01.479924 5675 log.cc:824] T c8f28bcf864d41a8a94b8bc4dacc1cb1 P 05fd30d5d3b84e93a6ec3a77870f6c34: Log is configured to *not* fsync() on all Append() calls
I20260619 18:47:01.480300 5676 tablet_bootstrap.cc:492] T c8f28bcf864d41a8a94b8bc4dacc1cb1 P 542019cd88ee453597eb659a94f7634b: Bootstrap starting.
I20260619 18:47:01.483654 5677 tablet_bootstrap.cc:492] T c8f28bcf864d41a8a94b8bc4dacc1cb1 P afeb1cbb5dbb4580ba7cf5183c0ae346: Bootstrap starting.
I20260619 18:47:01.484014 5676 tablet_bootstrap.cc:654] T c8f28bcf864d41a8a94b8bc4dacc1cb1 P 542019cd88ee453597eb659a94f7634b: Neither blocks nor log segments found. Creating new log.
I20260619 18:47:01.485203 5676 log.cc:824] T c8f28bcf864d41a8a94b8bc4dacc1cb1 P 542019cd88ee453597eb659a94f7634b: Log is configured to *not* fsync() on all Append() calls
I20260619 18:47:01.486301 5675 tablet_bootstrap.cc:492] T c8f28bcf864d41a8a94b8bc4dacc1cb1 P 05fd30d5d3b84e93a6ec3a77870f6c34: No bootstrap required, opened a new log
I20260619 18:47:01.486696 5675 ts_tablet_manager.cc:1397] T c8f28bcf864d41a8a94b8bc4dacc1cb1 P 05fd30d5d3b84e93a6ec3a77870f6c34: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.005s
I20260619 18:47:01.487221 5677 tablet_bootstrap.cc:654] T c8f28bcf864d41a8a94b8bc4dacc1cb1 P afeb1cbb5dbb4580ba7cf5183c0ae346: Neither blocks nor log segments found. Creating new log.
I20260619 18:47:01.488109 5677 log.cc:824] T c8f28bcf864d41a8a94b8bc4dacc1cb1 P afeb1cbb5dbb4580ba7cf5183c0ae346: Log is configured to *not* fsync() on all Append() calls
I20260619 18:47:01.488463 5676 tablet_bootstrap.cc:492] T c8f28bcf864d41a8a94b8bc4dacc1cb1 P 542019cd88ee453597eb659a94f7634b: No bootstrap required, opened a new log
I20260619 18:47:01.488700 5676 ts_tablet_manager.cc:1397] T c8f28bcf864d41a8a94b8bc4dacc1cb1 P 542019cd88ee453597eb659a94f7634b: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260619 18:47:01.490525 5677 tablet_bootstrap.cc:492] T c8f28bcf864d41a8a94b8bc4dacc1cb1 P afeb1cbb5dbb4580ba7cf5183c0ae346: No bootstrap required, opened a new log
I20260619 18:47:01.490759 5677 ts_tablet_manager.cc:1397] T c8f28bcf864d41a8a94b8bc4dacc1cb1 P afeb1cbb5dbb4580ba7cf5183c0ae346: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260619 18:47:01.496990 5675 raft_consensus.cc:348] T c8f28bcf864d41a8a94b8bc4dacc1cb1 P 05fd30d5d3b84e93a6ec3a77870f6c34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05fd30d5d3b84e93a6ec3a77870f6c34" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 39053 } } peers { permanent_uuid: "afeb1cbb5dbb4580ba7cf5183c0ae346" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 44041 } } peers { permanent_uuid: "542019cd88ee453597eb659a94f7634b" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 36729 } }
I20260619 18:47:01.497404 5675 raft_consensus.cc:374] T c8f28bcf864d41a8a94b8bc4dacc1cb1 P 05fd30d5d3b84e93a6ec3a77870f6c34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:47:01.497521 5675 raft_consensus.cc:729] T c8f28bcf864d41a8a94b8bc4dacc1cb1 P 05fd30d5d3b84e93a6ec3a77870f6c34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05fd30d5d3b84e93a6ec3a77870f6c34, State: Initialized, Role: FOLLOWER
I20260619 18:47:01.498003 5677 raft_consensus.cc:348] T c8f28bcf864d41a8a94b8bc4dacc1cb1 P afeb1cbb5dbb4580ba7cf5183c0ae346 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05fd30d5d3b84e93a6ec3a77870f6c34" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 39053 } } peers { permanent_uuid: "afeb1cbb5dbb4580ba7cf5183c0ae346" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 44041 } } peers { permanent_uuid: "542019cd88ee453597eb659a94f7634b" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 36729 } }
I20260619 18:47:01.498111 5675 consensus_queue.cc:260] T c8f28bcf864d41a8a94b8bc4dacc1cb1 P 05fd30d5d3b84e93a6ec3a77870f6c34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05fd30d5d3b84e93a6ec3a77870f6c34" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 39053 } } peers { permanent_uuid: "afeb1cbb5dbb4580ba7cf5183c0ae346" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 44041 } } peers { permanent_uuid: "542019cd88ee453597eb659a94f7634b" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 36729 } }
I20260619 18:47:01.498291 5677 raft_consensus.cc:374] T c8f28bcf864d41a8a94b8bc4dacc1cb1 P afeb1cbb5dbb4580ba7cf5183c0ae346 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:47:01.498416 5677 raft_consensus.cc:729] T c8f28bcf864d41a8a94b8bc4dacc1cb1 P afeb1cbb5dbb4580ba7cf5183c0ae346 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afeb1cbb5dbb4580ba7cf5183c0ae346, State: Initialized, Role: FOLLOWER
I20260619 18:47:01.498687 5676 raft_consensus.cc:348] T c8f28bcf864d41a8a94b8bc4dacc1cb1 P 542019cd88ee453597eb659a94f7634b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05fd30d5d3b84e93a6ec3a77870f6c34" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 39053 } } peers { permanent_uuid: "afeb1cbb5dbb4580ba7cf5183c0ae346" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 44041 } } peers { permanent_uuid: "542019cd88ee453597eb659a94f7634b" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 36729 } }
I20260619 18:47:01.499122 5676 raft_consensus.cc:374] T c8f28bcf864d41a8a94b8bc4dacc1cb1 P 542019cd88ee453597eb659a94f7634b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:47:01.499058 5677 consensus_queue.cc:260] T c8f28bcf864d41a8a94b8bc4dacc1cb1 P afeb1cbb5dbb4580ba7cf5183c0ae346 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05fd30d5d3b84e93a6ec3a77870f6c34" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 39053 } } peers { permanent_uuid: "afeb1cbb5dbb4580ba7cf5183c0ae346" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 44041 } } peers { permanent_uuid: "542019cd88ee453597eb659a94f7634b" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 36729 } }
I20260619 18:47:01.499249 5676 raft_consensus.cc:729] T c8f28bcf864d41a8a94b8bc4dacc1cb1 P 542019cd88ee453597eb659a94f7634b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 542019cd88ee453597eb659a94f7634b, State: Initialized, Role: FOLLOWER
I20260619 18:47:01.499926 5675 ts_tablet_manager.cc:1428] T c8f28bcf864d41a8a94b8bc4dacc1cb1 P 05fd30d5d3b84e93a6ec3a77870f6c34: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
I20260619 18:47:01.499827 5676 consensus_queue.cc:260] T c8f28bcf864d41a8a94b8bc4dacc1cb1 P 542019cd88ee453597eb659a94f7634b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05fd30d5d3b84e93a6ec3a77870f6c34" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 39053 } } peers { permanent_uuid: "afeb1cbb5dbb4580ba7cf5183c0ae346" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 44041 } } peers { permanent_uuid: "542019cd88ee453597eb659a94f7634b" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 36729 } }
I20260619 18:47:01.501076 5682 raft_consensus.cc:482] T c8f28bcf864d41a8a94b8bc4dacc1cb1 P 05fd30d5d3b84e93a6ec3a77870f6c34 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:47:01.501344 5682 raft_consensus.cc:504] T c8f28bcf864d41a8a94b8bc4dacc1cb1 P 05fd30d5d3b84e93a6ec3a77870f6c34 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05fd30d5d3b84e93a6ec3a77870f6c34" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 39053 } } peers { permanent_uuid: "afeb1cbb5dbb4580ba7cf5183c0ae346" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 44041 } } peers { permanent_uuid: "542019cd88ee453597eb659a94f7634b" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 36729 } }
I20260619 18:47:01.507290 5517 heartbeater.cc:503] Master 127.3.243.126:42779 was elected leader, sending a full tablet report...
I20260619 18:47:01.508229 5676 ts_tablet_manager.cc:1428] T c8f28bcf864d41a8a94b8bc4dacc1cb1 P 542019cd88ee453597eb659a94f7634b: Time spent starting tablet: real 0.019s user 0.011s sys 0.007s
I20260619 18:47:01.511673 5682 leader_election.cc:288] T c8f28bcf864d41a8a94b8bc4dacc1cb1 P 05fd30d5d3b84e93a6ec3a77870f6c34 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers afeb1cbb5dbb4580ba7cf5183c0ae346 (127.3.243.67:44041), 542019cd88ee453597eb659a94f7634b (127.3.243.66:36729)
I20260619 18:47:01.513470 5656 heartbeater.cc:503] Master 127.3.243.126:42779 was elected leader, sending a full tablet report...
I20260619 18:47:01.514101 5677 ts_tablet_manager.cc:1428] T c8f28bcf864d41a8a94b8bc4dacc1cb1 P afeb1cbb5dbb4580ba7cf5183c0ae346: Time spent starting tablet: real 0.023s user 0.014s sys 0.008s
I20260619 18:47:01.519932 5673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:01.504392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.65:33985 (local address 127.3.243.67:44041)
0619 18:47:01.504589 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:01.504595 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:01.512908 (+ 8313us) server_negotiation.cc:407] Connection header received
0619 18:47:01.513173 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:01.513178 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:01.513252 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:01.513338 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:47:01.514164 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:01.515048 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:01.518404 (+ 3356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:01.519153 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:01.519274 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:01.519323 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:01.519656 (+ 333us) server_negotiation.cc:299] Negotiation successful
0619 18:47:01.519737 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56,"spinlock_wait_cycles":14976}
I20260619 18:47:01.520094 5686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:01.503880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.67:44041 (local address 127.3.243.65:33985)
0619 18:47:01.512789 (+ 8909us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:01.512811 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:01.513023 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:01.513522 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:01.513527 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:01.513551 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:47:01.513968 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:01.513982 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:01.515212 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:01.515217 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:01.517324 (+ 2107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:01.517334 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:01.519344 (+ 2010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:01.519350 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:01.519517 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:01.519550 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 18:47:01.519636 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 18:47:01.519759 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8775,"thread_start_us":12673,"threads_started":1}
I20260619 18:47:01.521068 5606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8f28bcf864d41a8a94b8bc4dacc1cb1" candidate_uuid: "05fd30d5d3b84e93a6ec3a77870f6c34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "afeb1cbb5dbb4580ba7cf5183c0ae346" is_pre_election: true
I20260619 18:47:01.521592 5606 raft_consensus.cc:2454] T c8f28bcf864d41a8a94b8bc4dacc1cb1 P afeb1cbb5dbb4580ba7cf5183c0ae346 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05fd30d5d3b84e93a6ec3a77870f6c34 in term 0.
I20260619 18:47:01.521867 5674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:01.512567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.65:48933 (local address 127.3.243.66:36729)
0619 18:47:01.512748 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:01.512754 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:01.516454 (+ 3700us) server_negotiation.cc:407] Connection header received
0619 18:47:01.516580 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:01.516585 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:01.516647 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:01.516729 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:47:01.517483 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:01.518261 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:01.519982 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:01.520970 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:01.521120 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:01.521184 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:01.521519 (+ 335us) server_negotiation.cc:299] Negotiation successful
0619 18:47:01.521638 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 18:47:01.521876 5685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:01.512087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.66:36729 (local address 127.3.243.65:48933)
0619 18:47:01.516338 (+ 4251us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:01.516355 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:01.516481 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:01.516868 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:01.516872 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:01.516900 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:47:01.517291 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:01.517303 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:01.518370 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:01.518373 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:01.519831 (+ 1458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:01.519844 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:01.521187 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:01.521193 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:01.521357 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:01.521389 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 18:47:01.521481 (+ 92us) client_negotiation.cc:240] Negotiation successful
0619 18:47:01.521596 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4096,"thread_start_us":669,"threads_started":1}
I20260619 18:47:01.522260 5259 leader_election.cc:302] T c8f28bcf864d41a8a94b8bc4dacc1cb1 P 05fd30d5d3b84e93a6ec3a77870f6c34 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05fd30d5d3b84e93a6ec3a77870f6c34, afeb1cbb5dbb4580ba7cf5183c0ae346; no voters:
I20260619 18:47:01.522581 5682 raft_consensus.cc:2781] T c8f28bcf864d41a8a94b8bc4dacc1cb1 P 05fd30d5d3b84e93a6ec3a77870f6c34 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:47:01.522696 5682 raft_consensus.cc:482] T c8f28bcf864d41a8a94b8bc4dacc1cb1 P 05fd30d5d3b84e93a6ec3a77870f6c34 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:47:01.522606 5472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8f28bcf864d41a8a94b8bc4dacc1cb1" candidate_uuid: "05fd30d5d3b84e93a6ec3a77870f6c34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "542019cd88ee453597eb659a94f7634b" is_pre_election: true
I20260619 18:47:01.522784 5682 raft_consensus.cc:3037] T c8f28bcf864d41a8a94b8bc4dacc1cb1 P 05fd30d5d3b84e93a6ec3a77870f6c34 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:47:01.523002 5472 raft_consensus.cc:2454] T c8f28bcf864d41a8a94b8bc4dacc1cb1 P 542019cd88ee453597eb659a94f7634b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05fd30d5d3b84e93a6ec3a77870f6c34 in term 0.
I20260619 18:47:01.525000 5682 raft_consensus.cc:504] T c8f28bcf864d41a8a94b8bc4dacc1cb1 P 05fd30d5d3b84e93a6ec3a77870f6c34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05fd30d5d3b84e93a6ec3a77870f6c34" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 39053 } } peers { permanent_uuid: "afeb1cbb5dbb4580ba7cf5183c0ae346" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 44041 } } peers { permanent_uuid: "542019cd88ee453597eb659a94f7634b" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 36729 } }
I20260619 18:47:01.525597 5682 leader_election.cc:288] T c8f28bcf864d41a8a94b8bc4dacc1cb1 P 05fd30d5d3b84e93a6ec3a77870f6c34 [CANDIDATE]: Term 1 election: Requested vote from peers afeb1cbb5dbb4580ba7cf5183c0ae346 (127.3.243.67:44041), 542019cd88ee453597eb659a94f7634b (127.3.243.66:36729)
I20260619 18:47:01.525856 5606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8f28bcf864d41a8a94b8bc4dacc1cb1" candidate_uuid: "05fd30d5d3b84e93a6ec3a77870f6c34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "afeb1cbb5dbb4580ba7cf5183c0ae346"
I20260619 18:47:01.526041 5606 raft_consensus.cc:3037] T c8f28bcf864d41a8a94b8bc4dacc1cb1 P afeb1cbb5dbb4580ba7cf5183c0ae346 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:47:01.526018 5472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8f28bcf864d41a8a94b8bc4dacc1cb1" candidate_uuid: "05fd30d5d3b84e93a6ec3a77870f6c34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "542019cd88ee453597eb659a94f7634b"
I20260619 18:47:01.526185 5472 raft_consensus.cc:3037] T c8f28bcf864d41a8a94b8bc4dacc1cb1 P 542019cd88ee453597eb659a94f7634b [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:47:01.529147 5606 raft_consensus.cc:2454] T c8f28bcf864d41a8a94b8bc4dacc1cb1 P afeb1cbb5dbb4580ba7cf5183c0ae346 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05fd30d5d3b84e93a6ec3a77870f6c34 in term 1.
I20260619 18:47:01.529147 5472 raft_consensus.cc:2454] T c8f28bcf864d41a8a94b8bc4dacc1cb1 P 542019cd88ee453597eb659a94f7634b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05fd30d5d3b84e93a6ec3a77870f6c34 in term 1.
I20260619 18:47:01.529496 5261 leader_election.cc:302] T c8f28bcf864d41a8a94b8bc4dacc1cb1 P 05fd30d5d3b84e93a6ec3a77870f6c34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05fd30d5d3b84e93a6ec3a77870f6c34, 542019cd88ee453597eb659a94f7634b; no voters:
I20260619 18:47:01.529737 5682 raft_consensus.cc:2781] T c8f28bcf864d41a8a94b8bc4dacc1cb1 P 05fd30d5d3b84e93a6ec3a77870f6c34 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:47:01.529863 5682 raft_consensus.cc:686] T c8f28bcf864d41a8a94b8bc4dacc1cb1 P 05fd30d5d3b84e93a6ec3a77870f6c34 [term 1 LEADER]: Becoming Leader. State: Replica: 05fd30d5d3b84e93a6ec3a77870f6c34, State: Running, Role: LEADER
I20260619 18:47:01.530472 5682 consensus_queue.cc:237] T c8f28bcf864d41a8a94b8bc4dacc1cb1 P 05fd30d5d3b84e93a6ec3a77870f6c34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05fd30d5d3b84e93a6ec3a77870f6c34" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 39053 } } peers { permanent_uuid: "afeb1cbb5dbb4580ba7cf5183c0ae346" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 44041 } } peers { permanent_uuid: "542019cd88ee453597eb659a94f7634b" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 36729 } }
I20260619 18:47:01.534818 5172 catalog_manager.cc:5495] T c8f28bcf864d41a8a94b8bc4dacc1cb1 P 05fd30d5d3b84e93a6ec3a77870f6c34 reported cstate change: term changed from 0 to 1, leader changed from <none> to 05fd30d5d3b84e93a6ec3a77870f6c34 (127.3.243.65). New cstate: current_term: 1 leader_uuid: "05fd30d5d3b84e93a6ec3a77870f6c34" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05fd30d5d3b84e93a6ec3a77870f6c34" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 39053 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "afeb1cbb5dbb4580ba7cf5183c0ae346" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 44041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "542019cd88ee453597eb659a94f7634b" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 36729 } health_report { overall_health: UNKNOWN } } }
I20260619 18:47:01.573060 5670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:01.567992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46710 (local address 127.3.243.65:39053)
0619 18:47:01.568217 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:01.568224 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:01.568298 (+ 74us) server_negotiation.cc:407] Connection header received
0619 18:47:01.568630 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:01.568634 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:01.568728 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:01.568847 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:47:01.569123 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:01.569786 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:01.570990 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:01.571549 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:01.571641 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:01.572007 (+ 366us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:47:01.572577 (+ 570us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:47:01.572677 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:01.572754 (+ 77us) server_negotiation.cc:299] Negotiation successful
0619 18:47:01.572814 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 18:47:01.580829 5606 raft_consensus.cc:1261] T c8f28bcf864d41a8a94b8bc4dacc1cb1 P afeb1cbb5dbb4580ba7cf5183c0ae346 [term 1 FOLLOWER]: Refusing update from remote peer 05fd30d5d3b84e93a6ec3a77870f6c34: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:47:01.580829 5472 raft_consensus.cc:1261] T c8f28bcf864d41a8a94b8bc4dacc1cb1 P 542019cd88ee453597eb659a94f7634b [term 1 FOLLOWER]: Refusing update from remote peer 05fd30d5d3b84e93a6ec3a77870f6c34: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:47:01.581843 5681 consensus_queue.cc:1035] T c8f28bcf864d41a8a94b8bc4dacc1cb1 P 05fd30d5d3b84e93a6ec3a77870f6c34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "542019cd88ee453597eb659a94f7634b" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 36729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:47:01.582168 5682 consensus_queue.cc:1035] T c8f28bcf864d41a8a94b8bc4dacc1cb1 P 05fd30d5d3b84e93a6ec3a77870f6c34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "afeb1cbb5dbb4580ba7cf5183c0ae346" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 44041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:47:01.596614 5690 mvcc.cc:205] Tried to move back new op lower bound from 7298641189185056768 to 7298641188992126976. Current Snapshot: MvccSnapshot[applied={T|T < 7298641189185056768}]
I20260619 18:47:01.610603 5172 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41108:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:47:01.610836 5172 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41108:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:47:01.614132 5172 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9545518c740f4e57ab5c31e719e49d0c: Sending DeleteTablet for 3 replicas of tablet c8f28bcf864d41a8a94b8bc4dacc1cb1
I20260619 18:47:01.615155 5586 tablet_service.cc:1514] Processing DeleteTablet for tablet c8f28bcf864d41a8a94b8bc4dacc1cb1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:47:01 UTC) from {username='kudu', principal='kudu/127.3.243.126@KRBTEST.COM'} at 127.0.0.1:42102
I20260619 18:47:01.615167 5308 tablet_service.cc:1514] Processing DeleteTablet for tablet c8f28bcf864d41a8a94b8bc4dacc1cb1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:47:01 UTC) from {username='kudu', principal='kudu/127.3.243.126@KRBTEST.COM'} at 127.0.0.1:46702
I20260619 18:47:01.616091 5699 tablet_replica.cc:330] T c8f28bcf864d41a8a94b8bc4dacc1cb1 P afeb1cbb5dbb4580ba7cf5183c0ae346: stopping tablet replica
I20260619 18:47:01.616401 5698 tablet_replica.cc:330] T c8f28bcf864d41a8a94b8bc4dacc1cb1 P 05fd30d5d3b84e93a6ec3a77870f6c34: stopping tablet replica
I20260619 18:47:01.616787 5698 raft_consensus.cc:2229] T c8f28bcf864d41a8a94b8bc4dacc1cb1 P 05fd30d5d3b84e93a6ec3a77870f6c34 [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:47:01.616927 5699 raft_consensus.cc:2229] T c8f28bcf864d41a8a94b8bc4dacc1cb1 P afeb1cbb5dbb4580ba7cf5183c0ae346 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:47:01.616917 5447 tablet_service.cc:1514] Processing DeleteTablet for tablet c8f28bcf864d41a8a94b8bc4dacc1cb1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:47:01 UTC) from {username='kudu', principal='kudu/127.3.243.126@KRBTEST.COM'} at 127.0.0.1:51218
I20260619 18:47:01.617416 5699 raft_consensus.cc:2258] T c8f28bcf864d41a8a94b8bc4dacc1cb1 P afeb1cbb5dbb4580ba7cf5183c0ae346 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:47:01.617602 5698 raft_consensus.cc:2258] T c8f28bcf864d41a8a94b8bc4dacc1cb1 P 05fd30d5d3b84e93a6ec3a77870f6c34 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:47:01.618536 5700 tablet_replica.cc:330] T c8f28bcf864d41a8a94b8bc4dacc1cb1 P 542019cd88ee453597eb659a94f7634b: stopping tablet replica
I20260619 18:47:01.618932 5700 raft_consensus.cc:2229] T c8f28bcf864d41a8a94b8bc4dacc1cb1 P 542019cd88ee453597eb659a94f7634b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:47:01.619000 5172 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41108:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 18:47:01.619391 5699 ts_tablet_manager.cc:1905] T c8f28bcf864d41a8a94b8bc4dacc1cb1 P afeb1cbb5dbb4580ba7cf5183c0ae346: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260619 18:47:01.619577 5172 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:47:01.619599 5700 raft_consensus.cc:2258] T c8f28bcf864d41a8a94b8bc4dacc1cb1 P 542019cd88ee453597eb659a94f7634b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:47:01.622689 5698 ts_tablet_manager.cc:1905] T c8f28bcf864d41a8a94b8bc4dacc1cb1 P 05fd30d5d3b84e93a6ec3a77870f6c34: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:47:01.626264 5699 ts_tablet_manager.cc:1918] T c8f28bcf864d41a8a94b8bc4dacc1cb1 P afeb1cbb5dbb4580ba7cf5183c0ae346: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:47:01.626499 5699 log.cc:1196] T c8f28bcf864d41a8a94b8bc4dacc1cb1 P afeb1cbb5dbb4580ba7cf5183c0ae346: Deleting WAL directory at /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894810715282-4045-0/minicluster-data/ts-2/wal/wals/c8f28bcf864d41a8a94b8bc4dacc1cb1
I20260619 18:47:01.627116 5699 ts_tablet_manager.cc:1939] T c8f28bcf864d41a8a94b8bc4dacc1cb1 P afeb1cbb5dbb4580ba7cf5183c0ae346: Deleting consensus metadata
I20260619 18:47:01.627187 5308 tablet_service.cc:1467] Processing CreateTablet for tablet 593ee7cb4c744936af3fbc683f80b9ef (DEFAULT_TABLE table=test-table [id=c7c90aa6565042a2a16861c79e5cdcaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:47:01.627635 5308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 593ee7cb4c744936af3fbc683f80b9ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:47:01.628479 5447 tablet_service.cc:1467] Processing CreateTablet for tablet 593ee7cb4c744936af3fbc683f80b9ef (DEFAULT_TABLE table=test-table [id=c7c90aa6565042a2a16861c79e5cdcaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:47:01.628860 5447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 593ee7cb4c744936af3fbc683f80b9ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:47:01.630002 5698 ts_tablet_manager.cc:1918] T c8f28bcf864d41a8a94b8bc4dacc1cb1 P 05fd30d5d3b84e93a6ec3a77870f6c34: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:47:01.630141 5698 log.cc:1196] T c8f28bcf864d41a8a94b8bc4dacc1cb1 P 05fd30d5d3b84e93a6ec3a77870f6c34: Deleting WAL directory at /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894810715282-4045-0/minicluster-data/ts-0/wal/wals/c8f28bcf864d41a8a94b8bc4dacc1cb1
I20260619 18:47:01.630607 5698 ts_tablet_manager.cc:1939] T c8f28bcf864d41a8a94b8bc4dacc1cb1 P 05fd30d5d3b84e93a6ec3a77870f6c34: Deleting consensus metadata
I20260619 18:47:01.632206 5157 catalog_manager.cc:4841] TS 05fd30d5d3b84e93a6ec3a77870f6c34 (127.3.243.65:39053): tablet c8f28bcf864d41a8a94b8bc4dacc1cb1 (table test-table [id=2ceb3108526547bb8e338ed702657fbb]) successfully deleted
I20260619 18:47:01.632427 5586 tablet_service.cc:1467] Processing CreateTablet for tablet 593ee7cb4c744936af3fbc683f80b9ef (DEFAULT_TABLE table=test-table [id=c7c90aa6565042a2a16861c79e5cdcaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:47:01.632730 5586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 593ee7cb4c744936af3fbc683f80b9ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:47:01.636467 5156 catalog_manager.cc:4841] TS afeb1cbb5dbb4580ba7cf5183c0ae346 (127.3.243.67:44041): tablet c8f28bcf864d41a8a94b8bc4dacc1cb1 (table test-table [id=2ceb3108526547bb8e338ed702657fbb]) successfully deleted
I20260619 18:47:01.637086 5677 tablet_bootstrap.cc:492] T 593ee7cb4c744936af3fbc683f80b9ef P afeb1cbb5dbb4580ba7cf5183c0ae346: Bootstrap starting.
I20260619 18:47:01.638569 5677 tablet_bootstrap.cc:654] T 593ee7cb4c744936af3fbc683f80b9ef P afeb1cbb5dbb4580ba7cf5183c0ae346: Neither blocks nor log segments found. Creating new log.
I20260619 18:47:01.638995 5676 tablet_bootstrap.cc:492] T 593ee7cb4c744936af3fbc683f80b9ef P 542019cd88ee453597eb659a94f7634b: Bootstrap starting.
I20260619 18:47:01.640518 5676 tablet_bootstrap.cc:654] T 593ee7cb4c744936af3fbc683f80b9ef P 542019cd88ee453597eb659a94f7634b: Neither blocks nor log segments found. Creating new log.
I20260619 18:47:01.641381 5700 ts_tablet_manager.cc:1905] T c8f28bcf864d41a8a94b8bc4dacc1cb1 P 542019cd88ee453597eb659a94f7634b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:47:01.642130 5675 tablet_bootstrap.cc:492] T 593ee7cb4c744936af3fbc683f80b9ef P 05fd30d5d3b84e93a6ec3a77870f6c34: Bootstrap starting.
I20260619 18:47:01.643743 5675 tablet_bootstrap.cc:654] T 593ee7cb4c744936af3fbc683f80b9ef P 05fd30d5d3b84e93a6ec3a77870f6c34: Neither blocks nor log segments found. Creating new log.
I20260619 18:47:01.648969 5677 tablet_bootstrap.cc:492] T 593ee7cb4c744936af3fbc683f80b9ef P afeb1cbb5dbb4580ba7cf5183c0ae346: No bootstrap required, opened a new log
I20260619 18:47:01.649161 5677 ts_tablet_manager.cc:1397] T 593ee7cb4c744936af3fbc683f80b9ef P afeb1cbb5dbb4580ba7cf5183c0ae346: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.001s
I20260619 18:47:01.649960 5677 raft_consensus.cc:348] T 593ee7cb4c744936af3fbc683f80b9ef P afeb1cbb5dbb4580ba7cf5183c0ae346 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05fd30d5d3b84e93a6ec3a77870f6c34" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 39053 } } peers { permanent_uuid: "542019cd88ee453597eb659a94f7634b" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 36729 } } peers { permanent_uuid: "afeb1cbb5dbb4580ba7cf5183c0ae346" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 44041 } }
I20260619 18:47:01.650264 5677 raft_consensus.cc:374] T 593ee7cb4c744936af3fbc683f80b9ef P afeb1cbb5dbb4580ba7cf5183c0ae346 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:47:01.650403 5677 raft_consensus.cc:729] T 593ee7cb4c744936af3fbc683f80b9ef P afeb1cbb5dbb4580ba7cf5183c0ae346 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afeb1cbb5dbb4580ba7cf5183c0ae346, State: Initialized, Role: FOLLOWER
I20260619 18:47:01.650749 5677 consensus_queue.cc:260] T 593ee7cb4c744936af3fbc683f80b9ef P afeb1cbb5dbb4580ba7cf5183c0ae346 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05fd30d5d3b84e93a6ec3a77870f6c34" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 39053 } } peers { permanent_uuid: "542019cd88ee453597eb659a94f7634b" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 36729 } } peers { permanent_uuid: "afeb1cbb5dbb4580ba7cf5183c0ae346" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 44041 } }
I20260619 18:47:01.652727 5700 ts_tablet_manager.cc:1918] T c8f28bcf864d41a8a94b8bc4dacc1cb1 P 542019cd88ee453597eb659a94f7634b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:47:01.652946 5700 log.cc:1196] T c8f28bcf864d41a8a94b8bc4dacc1cb1 P 542019cd88ee453597eb659a94f7634b: Deleting WAL directory at /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894810715282-4045-0/minicluster-data/ts-1/wal/wals/c8f28bcf864d41a8a94b8bc4dacc1cb1
I20260619 18:47:01.653378 5676 tablet_bootstrap.cc:492] T 593ee7cb4c744936af3fbc683f80b9ef P 542019cd88ee453597eb659a94f7634b: No bootstrap required, opened a new log
I20260619 18:47:01.653582 5676 ts_tablet_manager.cc:1397] T 593ee7cb4c744936af3fbc683f80b9ef P 542019cd88ee453597eb659a94f7634b: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.003s
I20260619 18:47:01.653612 5677 ts_tablet_manager.cc:1428] T 593ee7cb4c744936af3fbc683f80b9ef P afeb1cbb5dbb4580ba7cf5183c0ae346: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20260619 18:47:01.653638 5700 ts_tablet_manager.cc:1939] T c8f28bcf864d41a8a94b8bc4dacc1cb1 P 542019cd88ee453597eb659a94f7634b: Deleting consensus metadata
I20260619 18:47:01.654492 5676 raft_consensus.cc:348] T 593ee7cb4c744936af3fbc683f80b9ef P 542019cd88ee453597eb659a94f7634b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05fd30d5d3b84e93a6ec3a77870f6c34" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 39053 } } peers { permanent_uuid: "542019cd88ee453597eb659a94f7634b" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 36729 } } peers { permanent_uuid: "afeb1cbb5dbb4580ba7cf5183c0ae346" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 44041 } }
I20260619 18:47:01.654808 5676 raft_consensus.cc:374] T 593ee7cb4c744936af3fbc683f80b9ef P 542019cd88ee453597eb659a94f7634b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:47:01.654949 5676 raft_consensus.cc:729] T 593ee7cb4c744936af3fbc683f80b9ef P 542019cd88ee453597eb659a94f7634b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 542019cd88ee453597eb659a94f7634b, State: Initialized, Role: FOLLOWER
I20260619 18:47:01.655292 5676 consensus_queue.cc:260] T 593ee7cb4c744936af3fbc683f80b9ef P 542019cd88ee453597eb659a94f7634b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05fd30d5d3b84e93a6ec3a77870f6c34" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 39053 } } peers { permanent_uuid: "542019cd88ee453597eb659a94f7634b" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 36729 } } peers { permanent_uuid: "afeb1cbb5dbb4580ba7cf5183c0ae346" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 44041 } }
I20260619 18:47:01.655920 5676 ts_tablet_manager.cc:1428] T 593ee7cb4c744936af3fbc683f80b9ef P 542019cd88ee453597eb659a94f7634b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260619 18:47:01.655894 5158 catalog_manager.cc:4841] TS 542019cd88ee453597eb659a94f7634b (127.3.243.66:36729): tablet c8f28bcf864d41a8a94b8bc4dacc1cb1 (table test-table [id=2ceb3108526547bb8e338ed702657fbb]) successfully deleted
I20260619 18:47:01.658027 5675 tablet_bootstrap.cc:492] T 593ee7cb4c744936af3fbc683f80b9ef P 05fd30d5d3b84e93a6ec3a77870f6c34: No bootstrap required, opened a new log
I20260619 18:47:01.658174 5675 ts_tablet_manager.cc:1397] T 593ee7cb4c744936af3fbc683f80b9ef P 05fd30d5d3b84e93a6ec3a77870f6c34: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.006s
I20260619 18:47:01.658977 5675 raft_consensus.cc:348] T 593ee7cb4c744936af3fbc683f80b9ef P 05fd30d5d3b84e93a6ec3a77870f6c34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05fd30d5d3b84e93a6ec3a77870f6c34" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 39053 } } peers { permanent_uuid: "542019cd88ee453597eb659a94f7634b" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 36729 } } peers { permanent_uuid: "afeb1cbb5dbb4580ba7cf5183c0ae346" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 44041 } }
I20260619 18:47:01.659251 5675 raft_consensus.cc:374] T 593ee7cb4c744936af3fbc683f80b9ef P 05fd30d5d3b84e93a6ec3a77870f6c34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:47:01.659400 5675 raft_consensus.cc:729] T 593ee7cb4c744936af3fbc683f80b9ef P 05fd30d5d3b84e93a6ec3a77870f6c34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05fd30d5d3b84e93a6ec3a77870f6c34, State: Initialized, Role: FOLLOWER
I20260619 18:47:01.659649 5675 consensus_queue.cc:260] T 593ee7cb4c744936af3fbc683f80b9ef P 05fd30d5d3b84e93a6ec3a77870f6c34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05fd30d5d3b84e93a6ec3a77870f6c34" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 39053 } } peers { permanent_uuid: "542019cd88ee453597eb659a94f7634b" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 36729 } } peers { permanent_uuid: "afeb1cbb5dbb4580ba7cf5183c0ae346" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 44041 } }
I20260619 18:47:01.660135 5675 ts_tablet_manager.cc:1428] T 593ee7cb4c744936af3fbc683f80b9ef P 05fd30d5d3b84e93a6ec3a77870f6c34: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260619 18:47:01.679546 5682 raft_consensus.cc:482] T 593ee7cb4c744936af3fbc683f80b9ef P 05fd30d5d3b84e93a6ec3a77870f6c34 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:47:01.679756 5682 raft_consensus.cc:504] T 593ee7cb4c744936af3fbc683f80b9ef P 05fd30d5d3b84e93a6ec3a77870f6c34 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05fd30d5d3b84e93a6ec3a77870f6c34" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 39053 } } peers { permanent_uuid: "542019cd88ee453597eb659a94f7634b" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 36729 } } peers { permanent_uuid: "afeb1cbb5dbb4580ba7cf5183c0ae346" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 44041 } }
I20260619 18:47:01.680501 5682 leader_election.cc:288] T 593ee7cb4c744936af3fbc683f80b9ef P 05fd30d5d3b84e93a6ec3a77870f6c34 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 542019cd88ee453597eb659a94f7634b (127.3.243.66:36729), afeb1cbb5dbb4580ba7cf5183c0ae346 (127.3.243.67:44041)
I20260619 18:47:01.681015 5606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "593ee7cb4c744936af3fbc683f80b9ef" candidate_uuid: "05fd30d5d3b84e93a6ec3a77870f6c34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "afeb1cbb5dbb4580ba7cf5183c0ae346" is_pre_election: true
I20260619 18:47:01.681017 5471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "593ee7cb4c744936af3fbc683f80b9ef" candidate_uuid: "05fd30d5d3b84e93a6ec3a77870f6c34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "542019cd88ee453597eb659a94f7634b" is_pre_election: true
I20260619 18:47:01.681254 5606 raft_consensus.cc:2454] T 593ee7cb4c744936af3fbc683f80b9ef P afeb1cbb5dbb4580ba7cf5183c0ae346 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05fd30d5d3b84e93a6ec3a77870f6c34 in term 0.
I20260619 18:47:01.681332 5471 raft_consensus.cc:2454] T 593ee7cb4c744936af3fbc683f80b9ef P 542019cd88ee453597eb659a94f7634b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05fd30d5d3b84e93a6ec3a77870f6c34 in term 0.
I20260619 18:47:01.681679 5259 leader_election.cc:302] T 593ee7cb4c744936af3fbc683f80b9ef P 05fd30d5d3b84e93a6ec3a77870f6c34 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05fd30d5d3b84e93a6ec3a77870f6c34, afeb1cbb5dbb4580ba7cf5183c0ae346; no voters:
I20260619 18:47:01.681952 5682 raft_consensus.cc:2781] T 593ee7cb4c744936af3fbc683f80b9ef P 05fd30d5d3b84e93a6ec3a77870f6c34 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:47:01.682065 5682 raft_consensus.cc:482] T 593ee7cb4c744936af3fbc683f80b9ef P 05fd30d5d3b84e93a6ec3a77870f6c34 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:47:01.682168 5682 raft_consensus.cc:3037] T 593ee7cb4c744936af3fbc683f80b9ef P 05fd30d5d3b84e93a6ec3a77870f6c34 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:47:01.684954 5682 raft_consensus.cc:504] T 593ee7cb4c744936af3fbc683f80b9ef P 05fd30d5d3b84e93a6ec3a77870f6c34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05fd30d5d3b84e93a6ec3a77870f6c34" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 39053 } } peers { permanent_uuid: "542019cd88ee453597eb659a94f7634b" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 36729 } } peers { permanent_uuid: "afeb1cbb5dbb4580ba7cf5183c0ae346" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 44041 } }
I20260619 18:47:01.685470 5682 leader_election.cc:288] T 593ee7cb4c744936af3fbc683f80b9ef P 05fd30d5d3b84e93a6ec3a77870f6c34 [CANDIDATE]: Term 1 election: Requested vote from peers 542019cd88ee453597eb659a94f7634b (127.3.243.66:36729), afeb1cbb5dbb4580ba7cf5183c0ae346 (127.3.243.67:44041)
I20260619 18:47:01.685938 5471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "593ee7cb4c744936af3fbc683f80b9ef" candidate_uuid: "05fd30d5d3b84e93a6ec3a77870f6c34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "542019cd88ee453597eb659a94f7634b"
I20260619 18:47:01.686010 5606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "593ee7cb4c744936af3fbc683f80b9ef" candidate_uuid: "05fd30d5d3b84e93a6ec3a77870f6c34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "afeb1cbb5dbb4580ba7cf5183c0ae346"
I20260619 18:47:01.686146 5471 raft_consensus.cc:3037] T 593ee7cb4c744936af3fbc683f80b9ef P 542019cd88ee453597eb659a94f7634b [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:47:01.686208 5606 raft_consensus.cc:3037] T 593ee7cb4c744936af3fbc683f80b9ef P afeb1cbb5dbb4580ba7cf5183c0ae346 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:47:01.689098 5471 raft_consensus.cc:2454] T 593ee7cb4c744936af3fbc683f80b9ef P 542019cd88ee453597eb659a94f7634b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05fd30d5d3b84e93a6ec3a77870f6c34 in term 1.
I20260619 18:47:01.689098 5606 raft_consensus.cc:2454] T 593ee7cb4c744936af3fbc683f80b9ef P afeb1cbb5dbb4580ba7cf5183c0ae346 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05fd30d5d3b84e93a6ec3a77870f6c34 in term 1.
I20260619 18:47:01.689514 5259 leader_election.cc:302] T 593ee7cb4c744936af3fbc683f80b9ef P 05fd30d5d3b84e93a6ec3a77870f6c34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05fd30d5d3b84e93a6ec3a77870f6c34, afeb1cbb5dbb4580ba7cf5183c0ae346; no voters:
I20260619 18:47:01.689833 5682 raft_consensus.cc:2781] T 593ee7cb4c744936af3fbc683f80b9ef P 05fd30d5d3b84e93a6ec3a77870f6c34 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:47:01.689981 5682 raft_consensus.cc:686] T 593ee7cb4c744936af3fbc683f80b9ef P 05fd30d5d3b84e93a6ec3a77870f6c34 [term 1 LEADER]: Becoming Leader. State: Replica: 05fd30d5d3b84e93a6ec3a77870f6c34, State: Running, Role: LEADER
I20260619 18:47:01.690291 5682 consensus_queue.cc:237] T 593ee7cb4c744936af3fbc683f80b9ef P 05fd30d5d3b84e93a6ec3a77870f6c34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05fd30d5d3b84e93a6ec3a77870f6c34" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 39053 } } peers { permanent_uuid: "542019cd88ee453597eb659a94f7634b" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 36729 } } peers { permanent_uuid: "afeb1cbb5dbb4580ba7cf5183c0ae346" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 44041 } }
I20260619 18:47:01.693126 5172 catalog_manager.cc:5495] T 593ee7cb4c744936af3fbc683f80b9ef P 05fd30d5d3b84e93a6ec3a77870f6c34 reported cstate change: term changed from 0 to 1, leader changed from <none> to 05fd30d5d3b84e93a6ec3a77870f6c34 (127.3.243.65). New cstate: current_term: 1 leader_uuid: "05fd30d5d3b84e93a6ec3a77870f6c34" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05fd30d5d3b84e93a6ec3a77870f6c34" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 39053 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "542019cd88ee453597eb659a94f7634b" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 36729 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "afeb1cbb5dbb4580ba7cf5183c0ae346" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 44041 } health_report { overall_health: UNKNOWN } } }
Jun 19 18:47:01 dist-test-slave-m9wg krb5kdc[5127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894819, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.126@KRBTEST.COM for kudu/127.3.243.126@KRBTEST.COM
I20260619 18:47:01.745894 5710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:01.730547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:42779 (local address 127.0.0.1:41112)
0619 18:47:01.731204 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:01.731223 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:01.731383 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:01.731765 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:01.731770 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:01.732786 (+ 1016us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:47:01.733113 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:01.733126 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:01.734207 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:01.734213 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:01.735756 (+ 1543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:01.735771 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:01.736574 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:01.736581 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:01.736754 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:01.737416 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:47:01.737440 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:47:01.739908 (+ 2468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:47:01.743552 (+ 3644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:01.743562 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:01.743567 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:01.743987 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:01.744366 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:01.744369 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:01.744371 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:01.744500 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:01.745071 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:47:01.745076 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:47:01.745228 (+ 152us) client_negotiation.cc:769] Sending connection context
0619 18:47:01.745353 (+ 125us) client_negotiation.cc:240] Negotiation successful
0619 18:47:01.745557 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":342,"threads_started":1}
I20260619 18:47:01.746098 5220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:01.730529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41112 (local address 127.3.243.126:42779)
0619 18:47:01.730726 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:01.730732 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:01.731315 (+ 583us) server_negotiation.cc:407] Connection header received
0619 18:47:01.731541 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:01.731545 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:01.731606 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:01.731690 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:01.733281 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:01.734082 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:01.735904 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:01.736396 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:01.736489 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:01.740084 (+ 3595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:01.740089 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:01.740093 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:01.740126 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:01.743423 (+ 3297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:01.744095 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:01.744100 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:01.744104 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:01.744267 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:01.744600 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:01.744604 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:01.744607 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:01.744953 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:01.745084 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:01.745474 (+ 390us) server_negotiation.cc:299] Negotiation successful
0619 18:47:01.745700 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 18:47:01.750757 5172 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.3.243.126@KRBTEST.COM'} at 127.0.0.1:41112:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 18:47:01.751734 5172 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:47:01.758080 5308 tablet_service.cc:1467] Processing CreateTablet for tablet 4ac29a2eea8e429e8aaac4df42a827df (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d82b475879c942da9d0a615d58720514]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 18:47:01.758201 5447 tablet_service.cc:1467] Processing CreateTablet for tablet 4ac29a2eea8e429e8aaac4df42a827df (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d82b475879c942da9d0a615d58720514]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 18:47:01.758472 5586 tablet_service.cc:1467] Processing CreateTablet for tablet 4ac29a2eea8e429e8aaac4df42a827df (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d82b475879c942da9d0a615d58720514]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 18:47:01.758672 5308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ac29a2eea8e429e8aaac4df42a827df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:47:01.758718 5447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ac29a2eea8e429e8aaac4df42a827df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:47:01.758948 5586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ac29a2eea8e429e8aaac4df42a827df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:47:01.765661 5677 tablet_bootstrap.cc:492] T 4ac29a2eea8e429e8aaac4df42a827df P afeb1cbb5dbb4580ba7cf5183c0ae346: Bootstrap starting.
I20260619 18:47:01.765718 5676 tablet_bootstrap.cc:492] T 4ac29a2eea8e429e8aaac4df42a827df P 542019cd88ee453597eb659a94f7634b: Bootstrap starting.
I20260619 18:47:01.765998 5675 tablet_bootstrap.cc:492] T 4ac29a2eea8e429e8aaac4df42a827df P 05fd30d5d3b84e93a6ec3a77870f6c34: Bootstrap starting.
I20260619 18:47:01.767274 5676 tablet_bootstrap.cc:654] T 4ac29a2eea8e429e8aaac4df42a827df P 542019cd88ee453597eb659a94f7634b: Neither blocks nor log segments found. Creating new log.
I20260619 18:47:01.767274 5677 tablet_bootstrap.cc:654] T 4ac29a2eea8e429e8aaac4df42a827df P afeb1cbb5dbb4580ba7cf5183c0ae346: Neither blocks nor log segments found. Creating new log.
I20260619 18:47:01.767727 5675 tablet_bootstrap.cc:654] T 4ac29a2eea8e429e8aaac4df42a827df P 05fd30d5d3b84e93a6ec3a77870f6c34: Neither blocks nor log segments found. Creating new log.
I20260619 18:47:01.778414 5677 tablet_bootstrap.cc:492] T 4ac29a2eea8e429e8aaac4df42a827df P afeb1cbb5dbb4580ba7cf5183c0ae346: No bootstrap required, opened a new log
I20260619 18:47:01.778462 5675 tablet_bootstrap.cc:492] T 4ac29a2eea8e429e8aaac4df42a827df P 05fd30d5d3b84e93a6ec3a77870f6c34: No bootstrap required, opened a new log
I20260619 18:47:01.778618 5675 ts_tablet_manager.cc:1397] T 4ac29a2eea8e429e8aaac4df42a827df P 05fd30d5d3b84e93a6ec3a77870f6c34: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.005s
I20260619 18:47:01.778914 5677 ts_tablet_manager.cc:1397] T 4ac29a2eea8e429e8aaac4df42a827df P afeb1cbb5dbb4580ba7cf5183c0ae346: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.008s
I20260619 18:47:01.779384 5675 raft_consensus.cc:348] T 4ac29a2eea8e429e8aaac4df42a827df P 05fd30d5d3b84e93a6ec3a77870f6c34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05fd30d5d3b84e93a6ec3a77870f6c34" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 39053 } } peers { permanent_uuid: "542019cd88ee453597eb659a94f7634b" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 36729 } } peers { permanent_uuid: "afeb1cbb5dbb4580ba7cf5183c0ae346" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 44041 } }
I20260619 18:47:01.779639 5675 raft_consensus.cc:374] T 4ac29a2eea8e429e8aaac4df42a827df P 05fd30d5d3b84e93a6ec3a77870f6c34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:47:01.779726 5675 raft_consensus.cc:729] T 4ac29a2eea8e429e8aaac4df42a827df P 05fd30d5d3b84e93a6ec3a77870f6c34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05fd30d5d3b84e93a6ec3a77870f6c34, State: Initialized, Role: FOLLOWER
I20260619 18:47:01.779923 5677 raft_consensus.cc:348] T 4ac29a2eea8e429e8aaac4df42a827df P afeb1cbb5dbb4580ba7cf5183c0ae346 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05fd30d5d3b84e93a6ec3a77870f6c34" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 39053 } } peers { permanent_uuid: "542019cd88ee453597eb659a94f7634b" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 36729 } } peers { permanent_uuid: "afeb1cbb5dbb4580ba7cf5183c0ae346" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 44041 } }
I20260619 18:47:01.780005 5675 consensus_queue.cc:260] T 4ac29a2eea8e429e8aaac4df42a827df P 05fd30d5d3b84e93a6ec3a77870f6c34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05fd30d5d3b84e93a6ec3a77870f6c34" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 39053 } } peers { permanent_uuid: "542019cd88ee453597eb659a94f7634b" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 36729 } } peers { permanent_uuid: "afeb1cbb5dbb4580ba7cf5183c0ae346" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 44041 } }
I20260619 18:47:01.780238 5677 raft_consensus.cc:374] T 4ac29a2eea8e429e8aaac4df42a827df P afeb1cbb5dbb4580ba7cf5183c0ae346 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:47:01.780362 5677 raft_consensus.cc:729] T 4ac29a2eea8e429e8aaac4df42a827df P afeb1cbb5dbb4580ba7cf5183c0ae346 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afeb1cbb5dbb4580ba7cf5183c0ae346, State: Initialized, Role: FOLLOWER
I20260619 18:47:01.780525 5682 tablet_replica.cc:439] T 4ac29a2eea8e429e8aaac4df42a827df P 05fd30d5d3b84e93a6ec3a77870f6c34: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05fd30d5d3b84e93a6ec3a77870f6c34" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 39053 } } peers { permanent_uuid: "542019cd88ee453597eb659a94f7634b" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 36729 } } peers { permanent_uuid: "afeb1cbb5dbb4580ba7cf5183c0ae346" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 44041 } } }
I20260619 18:47:01.780835 5682 tablet_replica.cc:442] T 4ac29a2eea8e429e8aaac4df42a827df P 05fd30d5d3b84e93a6ec3a77870f6c34: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 18:47:01.780719 5677 consensus_queue.cc:260] T 4ac29a2eea8e429e8aaac4df42a827df P afeb1cbb5dbb4580ba7cf5183c0ae346 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05fd30d5d3b84e93a6ec3a77870f6c34" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 39053 } } peers { permanent_uuid: "542019cd88ee453597eb659a94f7634b" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 36729 } } peers { permanent_uuid: "afeb1cbb5dbb4580ba7cf5183c0ae346" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 44041 } }
I20260619 18:47:01.781131 5683 tablet_replica.cc:439] T 4ac29a2eea8e429e8aaac4df42a827df P afeb1cbb5dbb4580ba7cf5183c0ae346: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05fd30d5d3b84e93a6ec3a77870f6c34" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 39053 } } peers { permanent_uuid: "542019cd88ee453597eb659a94f7634b" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 36729 } } peers { permanent_uuid: "afeb1cbb5dbb4580ba7cf5183c0ae346" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 44041 } } }
I20260619 18:47:01.781457 5683 tablet_replica.cc:442] T 4ac29a2eea8e429e8aaac4df42a827df P afeb1cbb5dbb4580ba7cf5183c0ae346: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 18:47:01.781790 5675 ts_tablet_manager.cc:1428] T 4ac29a2eea8e429e8aaac4df42a827df P 05fd30d5d3b84e93a6ec3a77870f6c34: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260619 18:47:01.783779 5676 tablet_bootstrap.cc:492] T 4ac29a2eea8e429e8aaac4df42a827df P 542019cd88ee453597eb659a94f7634b: No bootstrap required, opened a new log
I20260619 18:47:01.783999 5676 ts_tablet_manager.cc:1397] T 4ac29a2eea8e429e8aaac4df42a827df P 542019cd88ee453597eb659a94f7634b: Time spent bootstrapping tablet: real 0.018s user 0.001s sys 0.012s
I20260619 18:47:01.784475 5677 ts_tablet_manager.cc:1428] T 4ac29a2eea8e429e8aaac4df42a827df P afeb1cbb5dbb4580ba7cf5183c0ae346: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20260619 18:47:01.784864 5676 raft_consensus.cc:348] T 4ac29a2eea8e429e8aaac4df42a827df P 542019cd88ee453597eb659a94f7634b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05fd30d5d3b84e93a6ec3a77870f6c34" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 39053 } } peers { permanent_uuid: "542019cd88ee453597eb659a94f7634b" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 36729 } } peers { permanent_uuid: "afeb1cbb5dbb4580ba7cf5183c0ae346" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 44041 } }
I20260619 18:47:01.785109 5676 raft_consensus.cc:374] T 4ac29a2eea8e429e8aaac4df42a827df P 542019cd88ee453597eb659a94f7634b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:47:01.785185 5676 raft_consensus.cc:729] T 4ac29a2eea8e429e8aaac4df42a827df P 542019cd88ee453597eb659a94f7634b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 542019cd88ee453597eb659a94f7634b, State: Initialized, Role: FOLLOWER
I20260619 18:47:01.785425 5676 consensus_queue.cc:260] T 4ac29a2eea8e429e8aaac4df42a827df P 542019cd88ee453597eb659a94f7634b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05fd30d5d3b84e93a6ec3a77870f6c34" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 39053 } } peers { permanent_uuid: "542019cd88ee453597eb659a94f7634b" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 36729 } } peers { permanent_uuid: "afeb1cbb5dbb4580ba7cf5183c0ae346" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 44041 } }
I20260619 18:47:01.785944 5676 ts_tablet_manager.cc:1428] T 4ac29a2eea8e429e8aaac4df42a827df P 542019cd88ee453597eb659a94f7634b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260619 18:47:01.786103 5684 tablet_replica.cc:439] T 4ac29a2eea8e429e8aaac4df42a827df P 542019cd88ee453597eb659a94f7634b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05fd30d5d3b84e93a6ec3a77870f6c34" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 39053 } } peers { permanent_uuid: "542019cd88ee453597eb659a94f7634b" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 36729 } } peers { permanent_uuid: "afeb1cbb5dbb4580ba7cf5183c0ae346" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 44041 } } }
I20260619 18:47:01.786397 5684 tablet_replica.cc:442] T 4ac29a2eea8e429e8aaac4df42a827df P 542019cd88ee453597eb659a94f7634b: This TxnStatusTablet replica's current role is: FOLLOWER
W20260619 18:47:01.866955 5657 tablet.cc:2369] T 4ac29a2eea8e429e8aaac4df42a827df P afeb1cbb5dbb4580ba7cf5183c0ae346: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 18:47:01.871886 5518 tablet.cc:2369] T 4ac29a2eea8e429e8aaac4df42a827df P 542019cd88ee453597eb659a94f7634b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 18:47:01.891196 5379 tablet.cc:2369] T 4ac29a2eea8e429e8aaac4df42a827df P 05fd30d5d3b84e93a6ec3a77870f6c34: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:47:02.060631 5684 raft_consensus.cc:482] T 4ac29a2eea8e429e8aaac4df42a827df P 542019cd88ee453597eb659a94f7634b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:47:02.060881 5684 raft_consensus.cc:504] T 4ac29a2eea8e429e8aaac4df42a827df P 542019cd88ee453597eb659a94f7634b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05fd30d5d3b84e93a6ec3a77870f6c34" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 39053 } } peers { permanent_uuid: "542019cd88ee453597eb659a94f7634b" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 36729 } } peers { permanent_uuid: "afeb1cbb5dbb4580ba7cf5183c0ae346" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 44041 } }
I20260619 18:47:02.062359 5684 leader_election.cc:288] T 4ac29a2eea8e429e8aaac4df42a827df P 542019cd88ee453597eb659a94f7634b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05fd30d5d3b84e93a6ec3a77870f6c34 (127.3.243.65:39053), afeb1cbb5dbb4580ba7cf5183c0ae346 (127.3.243.67:44041)
I20260619 18:47:02.069257 5670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:02.062765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.66:33407 (local address 127.3.243.65:39053)
0619 18:47:02.062960 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:02.062967 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:02.063468 (+ 501us) server_negotiation.cc:407] Connection header received
0619 18:47:02.063815 (+ 347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:02.063819 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:02.063890 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:02.063971 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:47:02.064551 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:02.065499 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:02.067358 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:02.068395 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:02.068537 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:02.068609 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:02.068843 (+ 234us) server_negotiation.cc:299] Negotiation successful
0619 18:47:02.068970 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 18:47:02.069289 5716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:02.062765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.66:60485 (local address 127.3.243.67:44041)
0619 18:47:02.063632 (+ 867us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:02.063637 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:02.063961 (+ 324us) server_negotiation.cc:407] Connection header received
0619 18:47:02.064165 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:02.064171 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:02.064245 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:02.064340 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:47:02.064926 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:02.065820 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:02.067647 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:02.068402 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:02.068541 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:02.068609 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:02.068904 (+ 295us) server_negotiation.cc:299] Negotiation successful
0619 18:47:02.069032 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":732,"thread_start_us":616,"threads_started":1}
I20260619 18:47:02.069372 5714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:02.062707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.67:44041 (local address 127.3.243.66:60485)
0619 18:47:02.063862 (+ 1155us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:02.063874 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:02.063991 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:02.064374 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:02.064378 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:02.064401 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:47:02.064751 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:02.064763 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:02.065946 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:02.065950 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:02.067504 (+ 1554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:02.067517 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:02.068614 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:02.068620 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:02.068748 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:02.068769 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 18:47:02.068867 (+ 98us) client_negotiation.cc:240] Negotiation successful
0619 18:47:02.068983 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1067,"mutex_wait_us":18,"spinlock_wait_cycles":8448,"thread_start_us":386,"threads_started":1}
I20260619 18:47:02.069406 5715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:02.062675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.65:39053 (local address 127.3.243.66:33407)
0619 18:47:02.063357 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:02.063377 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:02.063613 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:02.064010 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:02.064014 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:02.064040 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:47:02.064332 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:02.064344 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:02.065621 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:02.065626 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:02.067200 (+ 1574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:02.067211 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:02.068533 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:02.068538 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:02.068705 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:02.068729 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 18:47:02.068804 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 18:47:02.068925 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"mutex_wait_us":62,"thread_start_us":923,"threads_started":1}
I20260619 18:47:02.069948 5606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ac29a2eea8e429e8aaac4df42a827df" candidate_uuid: "542019cd88ee453597eb659a94f7634b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "afeb1cbb5dbb4580ba7cf5183c0ae346" is_pre_election: true
I20260619 18:47:02.070204 5606 raft_consensus.cc:2454] T 4ac29a2eea8e429e8aaac4df42a827df P afeb1cbb5dbb4580ba7cf5183c0ae346 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 542019cd88ee453597eb659a94f7634b in term 0.
I20260619 18:47:02.070369 5328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ac29a2eea8e429e8aaac4df42a827df" candidate_uuid: "542019cd88ee453597eb659a94f7634b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05fd30d5d3b84e93a6ec3a77870f6c34" is_pre_election: true
I20260619 18:47:02.070647 5398 leader_election.cc:302] T 4ac29a2eea8e429e8aaac4df42a827df P 542019cd88ee453597eb659a94f7634b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 542019cd88ee453597eb659a94f7634b, afeb1cbb5dbb4580ba7cf5183c0ae346; no voters:
I20260619 18:47:02.070711 5328 raft_consensus.cc:2454] T 4ac29a2eea8e429e8aaac4df42a827df P 05fd30d5d3b84e93a6ec3a77870f6c34 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 542019cd88ee453597eb659a94f7634b in term 0.
I20260619 18:47:02.071029 5684 raft_consensus.cc:2781] T 4ac29a2eea8e429e8aaac4df42a827df P 542019cd88ee453597eb659a94f7634b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:47:02.071130 5684 raft_consensus.cc:482] T 4ac29a2eea8e429e8aaac4df42a827df P 542019cd88ee453597eb659a94f7634b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:47:02.071224 5684 raft_consensus.cc:3037] T 4ac29a2eea8e429e8aaac4df42a827df P 542019cd88ee453597eb659a94f7634b [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:47:02.073397 5684 raft_consensus.cc:504] T 4ac29a2eea8e429e8aaac4df42a827df P 542019cd88ee453597eb659a94f7634b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05fd30d5d3b84e93a6ec3a77870f6c34" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 39053 } } peers { permanent_uuid: "542019cd88ee453597eb659a94f7634b" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 36729 } } peers { permanent_uuid: "afeb1cbb5dbb4580ba7cf5183c0ae346" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 44041 } }
I20260619 18:47:02.073939 5684 leader_election.cc:288] T 4ac29a2eea8e429e8aaac4df42a827df P 542019cd88ee453597eb659a94f7634b [CANDIDATE]: Term 1 election: Requested vote from peers 05fd30d5d3b84e93a6ec3a77870f6c34 (127.3.243.65:39053), afeb1cbb5dbb4580ba7cf5183c0ae346 (127.3.243.67:44041)
I20260619 18:47:02.074283 5328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ac29a2eea8e429e8aaac4df42a827df" candidate_uuid: "542019cd88ee453597eb659a94f7634b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05fd30d5d3b84e93a6ec3a77870f6c34"
I20260619 18:47:02.074337 5606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ac29a2eea8e429e8aaac4df42a827df" candidate_uuid: "542019cd88ee453597eb659a94f7634b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "afeb1cbb5dbb4580ba7cf5183c0ae346"
I20260619 18:47:02.074453 5328 raft_consensus.cc:3037] T 4ac29a2eea8e429e8aaac4df42a827df P 05fd30d5d3b84e93a6ec3a77870f6c34 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:47:02.074535 5606 raft_consensus.cc:3037] T 4ac29a2eea8e429e8aaac4df42a827df P afeb1cbb5dbb4580ba7cf5183c0ae346 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:47:02.076555 5606 raft_consensus.cc:2454] T 4ac29a2eea8e429e8aaac4df42a827df P afeb1cbb5dbb4580ba7cf5183c0ae346 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 542019cd88ee453597eb659a94f7634b in term 1.
I20260619 18:47:02.076565 5328 raft_consensus.cc:2454] T 4ac29a2eea8e429e8aaac4df42a827df P 05fd30d5d3b84e93a6ec3a77870f6c34 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 542019cd88ee453597eb659a94f7634b in term 1.
I20260619 18:47:02.076879 5398 leader_election.cc:302] T 4ac29a2eea8e429e8aaac4df42a827df P 542019cd88ee453597eb659a94f7634b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 542019cd88ee453597eb659a94f7634b, afeb1cbb5dbb4580ba7cf5183c0ae346; no voters:
I20260619 18:47:02.077129 5684 raft_consensus.cc:2781] T 4ac29a2eea8e429e8aaac4df42a827df P 542019cd88ee453597eb659a94f7634b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:47:02.077494 5684 raft_consensus.cc:686] T 4ac29a2eea8e429e8aaac4df42a827df P 542019cd88ee453597eb659a94f7634b [term 1 LEADER]: Becoming Leader. State: Replica: 542019cd88ee453597eb659a94f7634b, State: Running, Role: LEADER
I20260619 18:47:02.077821 5684 consensus_queue.cc:237] T 4ac29a2eea8e429e8aaac4df42a827df P 542019cd88ee453597eb659a94f7634b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05fd30d5d3b84e93a6ec3a77870f6c34" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 39053 } } peers { permanent_uuid: "542019cd88ee453597eb659a94f7634b" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 36729 } } peers { permanent_uuid: "afeb1cbb5dbb4580ba7cf5183c0ae346" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 44041 } }
I20260619 18:47:02.079531 5717 tablet_replica.cc:439] T 4ac29a2eea8e429e8aaac4df42a827df P 542019cd88ee453597eb659a94f7634b: TxnStatusTablet state changed. Reason: New leader 542019cd88ee453597eb659a94f7634b. Latest consensus state: current_term: 1 leader_uuid: "542019cd88ee453597eb659a94f7634b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05fd30d5d3b84e93a6ec3a77870f6c34" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 39053 } } peers { permanent_uuid: "542019cd88ee453597eb659a94f7634b" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 36729 } } peers { permanent_uuid: "afeb1cbb5dbb4580ba7cf5183c0ae346" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 44041 } } }
I20260619 18:47:02.079910 5717 tablet_replica.cc:442] T 4ac29a2eea8e429e8aaac4df42a827df P 542019cd88ee453597eb659a94f7634b: This TxnStatusTablet replica's current role is: LEADER
I20260619 18:47:02.080655 5719 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 18:47:02.080863 5719 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 18:47:02.081141 5171 catalog_manager.cc:5495] T 4ac29a2eea8e429e8aaac4df42a827df P 542019cd88ee453597eb659a94f7634b reported cstate change: term changed from 0 to 1, leader changed from <none> to 542019cd88ee453597eb659a94f7634b (127.3.243.66). New cstate: current_term: 1 leader_uuid: "542019cd88ee453597eb659a94f7634b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05fd30d5d3b84e93a6ec3a77870f6c34" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 39053 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "542019cd88ee453597eb659a94f7634b" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 36729 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "afeb1cbb5dbb4580ba7cf5183c0ae346" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 44041 } health_report { overall_health: UNKNOWN } } }
I20260619 18:47:02.148955 5710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:02.143185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.66:36729 (local address 127.0.0.1:51230)
0619 18:47:02.143409 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:02.143430 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:02.143576 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:02.144226 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:02.144231 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:02.144243 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 18:47:02.144450 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:02.144459 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:02.145549 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:02.145555 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:02.146537 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:02.146549 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:02.147263 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:02.147269 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:02.147415 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:02.147472 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 18:47:02.148388 (+ 916us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 18:47:02.148411 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 18:47:02.148494 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 18:47:02.148658 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260619 18:47:02.149001 5722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:02.143279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51230 (local address 127.3.243.66:36729)
0619 18:47:02.143837 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:02.143841 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:02.143855 (+ 14us) server_negotiation.cc:407] Connection header received
0619 18:47:02.143931 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:02.143939 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:02.144016 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:02.144177 (+ 161us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:47:02.144600 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:02.145415 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:02.146668 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:02.147116 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:02.147220 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:02.147651 (+ 431us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:47:02.148285 (+ 634us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:47:02.148419 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:02.148547 (+ 128us) server_negotiation.cc:299] Negotiation successful
0619 18:47:02.148633 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":253,"threads_started":1}
I20260619 18:47:02.153427 5606 raft_consensus.cc:1261] T 4ac29a2eea8e429e8aaac4df42a827df P afeb1cbb5dbb4580ba7cf5183c0ae346 [term 1 FOLLOWER]: Refusing update from remote peer 542019cd88ee453597eb659a94f7634b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:47:02.153635 5328 raft_consensus.cc:1261] T 4ac29a2eea8e429e8aaac4df42a827df P 05fd30d5d3b84e93a6ec3a77870f6c34 [term 1 FOLLOWER]: Refusing update from remote peer 542019cd88ee453597eb659a94f7634b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:47:02.154206 5717 consensus_queue.cc:1035] T 4ac29a2eea8e429e8aaac4df42a827df P 542019cd88ee453597eb659a94f7634b [LEADER]: Connected to new peer: Peer: permanent_uuid: "afeb1cbb5dbb4580ba7cf5183c0ae346" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 44041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:47:02.154517 5684 consensus_queue.cc:1035] T 4ac29a2eea8e429e8aaac4df42a827df P 542019cd88ee453597eb659a94f7634b [LEADER]: Connected to new peer: Peer: permanent_uuid: "05fd30d5d3b84e93a6ec3a77870f6c34" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 39053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:47:02.160900 5683 tablet_replica.cc:439] T 4ac29a2eea8e429e8aaac4df42a827df P afeb1cbb5dbb4580ba7cf5183c0ae346: TxnStatusTablet state changed. Reason: New leader 542019cd88ee453597eb659a94f7634b. Latest consensus state: current_term: 1 leader_uuid: "542019cd88ee453597eb659a94f7634b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05fd30d5d3b84e93a6ec3a77870f6c34" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 39053 } } peers { permanent_uuid: "542019cd88ee453597eb659a94f7634b" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 36729 } } peers { permanent_uuid: "afeb1cbb5dbb4580ba7cf5183c0ae346" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 44041 } } }
I20260619 18:47:02.161183 5683 tablet_replica.cc:442] T 4ac29a2eea8e429e8aaac4df42a827df P afeb1cbb5dbb4580ba7cf5183c0ae346: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 18:47:02.164109 5684 tablet_replica.cc:439] T 4ac29a2eea8e429e8aaac4df42a827df P 542019cd88ee453597eb659a94f7634b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "542019cd88ee453597eb659a94f7634b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05fd30d5d3b84e93a6ec3a77870f6c34" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 39053 } } peers { permanent_uuid: "542019cd88ee453597eb659a94f7634b" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 36729 } } peers { permanent_uuid: "afeb1cbb5dbb4580ba7cf5183c0ae346" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 44041 } } }
I20260619 18:47:02.164476 5684 tablet_replica.cc:442] T 4ac29a2eea8e429e8aaac4df42a827df P 542019cd88ee453597eb659a94f7634b: This TxnStatusTablet replica's current role is: LEADER
I20260619 18:47:02.166882 5682 tablet_replica.cc:439] T 4ac29a2eea8e429e8aaac4df42a827df P 05fd30d5d3b84e93a6ec3a77870f6c34: TxnStatusTablet state changed. Reason: New leader 542019cd88ee453597eb659a94f7634b. Latest consensus state: current_term: 1 leader_uuid: "542019cd88ee453597eb659a94f7634b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05fd30d5d3b84e93a6ec3a77870f6c34" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 39053 } } peers { permanent_uuid: "542019cd88ee453597eb659a94f7634b" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 36729 } } peers { permanent_uuid: "afeb1cbb5dbb4580ba7cf5183c0ae346" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 44041 } } }
I20260619 18:47:02.167129 5682 tablet_replica.cc:442] T 4ac29a2eea8e429e8aaac4df42a827df P 05fd30d5d3b84e93a6ec3a77870f6c34: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 18:47:02.169011 5684 tablet_replica.cc:439] T 4ac29a2eea8e429e8aaac4df42a827df P 542019cd88ee453597eb659a94f7634b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "542019cd88ee453597eb659a94f7634b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05fd30d5d3b84e93a6ec3a77870f6c34" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 39053 } } peers { permanent_uuid: "542019cd88ee453597eb659a94f7634b" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 36729 } } peers { permanent_uuid: "afeb1cbb5dbb4580ba7cf5183c0ae346" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 44041 } } }
I20260619 18:47:02.171231 5684 tablet_replica.cc:442] T 4ac29a2eea8e429e8aaac4df42a827df P 542019cd88ee453597eb659a94f7634b: This TxnStatusTablet replica's current role is: LEADER
I20260619 18:47:02.175652 5683 tablet_replica.cc:439] T 4ac29a2eea8e429e8aaac4df42a827df P afeb1cbb5dbb4580ba7cf5183c0ae346: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "542019cd88ee453597eb659a94f7634b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05fd30d5d3b84e93a6ec3a77870f6c34" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 39053 } } peers { permanent_uuid: "542019cd88ee453597eb659a94f7634b" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 36729 } } peers { permanent_uuid: "afeb1cbb5dbb4580ba7cf5183c0ae346" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 44041 } } }
I20260619 18:47:02.175884 5683 tablet_replica.cc:442] T 4ac29a2eea8e429e8aaac4df42a827df P afeb1cbb5dbb4580ba7cf5183c0ae346: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 18:47:02.179612 5682 tablet_replica.cc:439] T 4ac29a2eea8e429e8aaac4df42a827df P 05fd30d5d3b84e93a6ec3a77870f6c34: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "542019cd88ee453597eb659a94f7634b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05fd30d5d3b84e93a6ec3a77870f6c34" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 39053 } } peers { permanent_uuid: "542019cd88ee453597eb659a94f7634b" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 36729 } } peers { permanent_uuid: "afeb1cbb5dbb4580ba7cf5183c0ae346" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 44041 } } }
I20260619 18:47:02.179883 5682 tablet_replica.cc:442] T 4ac29a2eea8e429e8aaac4df42a827df P 05fd30d5d3b84e93a6ec3a77870f6c34: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 18:47:02.189173 5722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:02.184079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.65:35833 (local address 127.3.243.66:36729)
0619 18:47:02.184253 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:02.184257 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:02.184751 (+ 494us) server_negotiation.cc:407] Connection header received
0619 18:47:02.184936 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:02.184940 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:02.185003 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:02.185078 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:47:02.185503 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:02.186213 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:02.187543 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:02.187955 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:02.188060 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:02.188484 (+ 424us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:47:02.188591 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:47:02.188663 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:02.188784 (+ 121us) server_negotiation.cc:299] Negotiation successful
0619 18:47:02.188858 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 18:47:02.189234 5733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:02.184065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.66:36729 (local address 127.3.243.65:35833)
0619 18:47:02.184654 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:02.184669 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:02.184807 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:02.185101 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:02.185105 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:02.185117 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 18:47:02.185369 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:02.185377 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:02.186342 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:02.186347 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:02.187416 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:02.187427 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:02.188089 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:02.188095 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:02.188262 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:02.188318 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 18:47:02.188679 (+ 361us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 18:47:02.188690 (+ 11us) client_negotiation.cc:769] Sending connection context
0619 18:47:02.188761 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 18:47:02.188895 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":316,"threads_started":1}
I20260619 18:47:02.194945 5682 consensus_queue.cc:1035] T 593ee7cb4c744936af3fbc683f80b9ef P 05fd30d5d3b84e93a6ec3a77870f6c34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "542019cd88ee453597eb659a94f7634b" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 36729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:47:02.208220 5606 raft_consensus.cc:1261] T 593ee7cb4c744936af3fbc683f80b9ef P afeb1cbb5dbb4580ba7cf5183c0ae346 [term 1 FOLLOWER]: Refusing update from remote peer 05fd30d5d3b84e93a6ec3a77870f6c34: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:47:02.210009 5737 consensus_queue.cc:1035] T 593ee7cb4c744936af3fbc683f80b9ef P 05fd30d5d3b84e93a6ec3a77870f6c34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "afeb1cbb5dbb4580ba7cf5183c0ae346" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 44041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:47:02.251562 5670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:02.245682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.66:41461 (local address 127.3.243.65:39053)
0619 18:47:02.245827 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:02.245831 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:02.246505 (+ 674us) server_negotiation.cc:407] Connection header received
0619 18:47:02.246817 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:02.246823 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:02.246894 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:02.246970 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:47:02.247567 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:02.248407 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:02.249808 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:02.250223 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:02.250325 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:02.250786 (+ 461us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:47:02.250908 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:47:02.250997 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:02.251211 (+ 214us) server_negotiation.cc:299] Negotiation successful
0619 18:47:02.251294 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 18:47:02.251711 5742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:02.245751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.65:39053 (local address 127.3.243.66:41461)
0619 18:47:02.246396 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:02.246419 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:02.246626 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:02.246997 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:02.247001 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:02.247012 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 18:47:02.247403 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:02.247413 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:02.248532 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:02.248537 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:02.249680 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:02.249694 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:02.250386 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:02.250393 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:02.250550 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:02.250621 (+ 71us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 18:47:02.251021 (+ 400us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 18:47:02.251035 (+ 14us) client_negotiation.cc:769] Sending connection context
0619 18:47:02.251166 (+ 131us) client_negotiation.cc:240] Negotiation successful
0619 18:47:02.251297 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":309,"threads_started":1}
I20260619 18:47:02.281899 5172 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41108:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:47:02.282207 5172 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41108:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:47:02.285212 5172 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9545518c740f4e57ab5c31e719e49d0c: Sending DeleteTablet for 3 replicas of tablet 593ee7cb4c744936af3fbc683f80b9ef
I20260619 18:47:02.285897 5308 tablet_service.cc:1514] Processing DeleteTablet for tablet 593ee7cb4c744936af3fbc683f80b9ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:47:02 UTC) from {username='kudu', principal='kudu/127.3.243.126@KRBTEST.COM'} at 127.0.0.1:46702
I20260619 18:47:02.286015 5447 tablet_service.cc:1514] Processing DeleteTablet for tablet 593ee7cb4c744936af3fbc683f80b9ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:47:02 UTC) from {username='kudu', principal='kudu/127.3.243.126@KRBTEST.COM'} at 127.0.0.1:51218
I20260619 18:47:02.286322 5586 tablet_service.cc:1514] Processing DeleteTablet for tablet 593ee7cb4c744936af3fbc683f80b9ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:47:02 UTC) from {username='kudu', principal='kudu/127.3.243.126@KRBTEST.COM'} at 127.0.0.1:42102
I20260619 18:47:02.286928 5745 tablet_replica.cc:330] T 593ee7cb4c744936af3fbc683f80b9ef P 542019cd88ee453597eb659a94f7634b: stopping tablet replica
I20260619 18:47:02.287099 5746 tablet_replica.cc:330] T 593ee7cb4c744936af3fbc683f80b9ef P afeb1cbb5dbb4580ba7cf5183c0ae346: stopping tablet replica
I20260619 18:47:02.287206 5745 raft_consensus.cc:2229] T 593ee7cb4c744936af3fbc683f80b9ef P 542019cd88ee453597eb659a94f7634b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:47:02.287319 5746 raft_consensus.cc:2229] T 593ee7cb4c744936af3fbc683f80b9ef P afeb1cbb5dbb4580ba7cf5183c0ae346 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:47:02.287606 5746 raft_consensus.cc:2258] T 593ee7cb4c744936af3fbc683f80b9ef P afeb1cbb5dbb4580ba7cf5183c0ae346 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:47:02.287576 5745 raft_consensus.cc:2258] T 593ee7cb4c744936af3fbc683f80b9ef P 542019cd88ee453597eb659a94f7634b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:47:02.288745 5745 ts_tablet_manager.cc:1905] T 593ee7cb4c744936af3fbc683f80b9ef P 542019cd88ee453597eb659a94f7634b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:47:02.288758 5746 ts_tablet_manager.cc:1905] T 593ee7cb4c744936af3fbc683f80b9ef P afeb1cbb5dbb4580ba7cf5183c0ae346: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:47:02.295516 5746 ts_tablet_manager.cc:1918] T 593ee7cb4c744936af3fbc683f80b9ef P afeb1cbb5dbb4580ba7cf5183c0ae346: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 18:47:02.295728 5746 log.cc:1196] T 593ee7cb4c744936af3fbc683f80b9ef P afeb1cbb5dbb4580ba7cf5183c0ae346: Deleting WAL directory at /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894810715282-4045-0/minicluster-data/ts-2/wal/wals/593ee7cb4c744936af3fbc683f80b9ef
I20260619 18:47:02.296276 5746 ts_tablet_manager.cc:1939] T 593ee7cb4c744936af3fbc683f80b9ef P afeb1cbb5dbb4580ba7cf5183c0ae346: Deleting consensus metadata
I20260619 18:47:02.296566 5745 ts_tablet_manager.cc:1918] T 593ee7cb4c744936af3fbc683f80b9ef P 542019cd88ee453597eb659a94f7634b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 18:47:02.296705 5745 log.cc:1196] T 593ee7cb4c744936af3fbc683f80b9ef P 542019cd88ee453597eb659a94f7634b: Deleting WAL directory at /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894810715282-4045-0/minicluster-data/ts-1/wal/wals/593ee7cb4c744936af3fbc683f80b9ef
I20260619 18:47:02.297135 5745 ts_tablet_manager.cc:1939] T 593ee7cb4c744936af3fbc683f80b9ef P 542019cd88ee453597eb659a94f7634b: Deleting consensus metadata
I20260619 18:47:02.297681 5156 catalog_manager.cc:4841] TS afeb1cbb5dbb4580ba7cf5183c0ae346 (127.3.243.67:44041): tablet 593ee7cb4c744936af3fbc683f80b9ef (table test-table [id=c7c90aa6565042a2a16861c79e5cdcaa]) successfully deleted
I20260619 18:47:02.298447 5158 catalog_manager.cc:4841] TS 542019cd88ee453597eb659a94f7634b (127.3.243.66:36729): tablet 593ee7cb4c744936af3fbc683f80b9ef (table test-table [id=c7c90aa6565042a2a16861c79e5cdcaa]) successfully deleted
I20260619 18:47:02.304806 5744 tablet_replica.cc:330] T 593ee7cb4c744936af3fbc683f80b9ef P 05fd30d5d3b84e93a6ec3a77870f6c34: stopping tablet replica
I20260619 18:47:02.305015 5744 raft_consensus.cc:2229] T 593ee7cb4c744936af3fbc683f80b9ef P 05fd30d5d3b84e93a6ec3a77870f6c34 [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:47:02.305375 5744 raft_consensus.cc:2258] T 593ee7cb4c744936af3fbc683f80b9ef P 05fd30d5d3b84e93a6ec3a77870f6c34 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:47:02.306298 5744 ts_tablet_manager.cc:1905] T 593ee7cb4c744936af3fbc683f80b9ef P 05fd30d5d3b84e93a6ec3a77870f6c34: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:47:02.312487 5744 ts_tablet_manager.cc:1918] T 593ee7cb4c744936af3fbc683f80b9ef P 05fd30d5d3b84e93a6ec3a77870f6c34: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 18:47:02.312624 5744 log.cc:1196] T 593ee7cb4c744936af3fbc683f80b9ef P 05fd30d5d3b84e93a6ec3a77870f6c34: Deleting WAL directory at /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894810715282-4045-0/minicluster-data/ts-0/wal/wals/593ee7cb4c744936af3fbc683f80b9ef
I20260619 18:47:02.313006 5744 ts_tablet_manager.cc:1939] T 593ee7cb4c744936af3fbc683f80b9ef P 05fd30d5d3b84e93a6ec3a77870f6c34: Deleting consensus metadata
I20260619 18:47:02.314134 5157 catalog_manager.cc:4841] TS 05fd30d5d3b84e93a6ec3a77870f6c34 (127.3.243.65:39053): tablet 593ee7cb4c744936af3fbc683f80b9ef (table test-table [id=c7c90aa6565042a2a16861c79e5cdcaa]) successfully deleted
Jun 19 18:47:02 dist-test-slave-m9wg krb5kdc[5127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894821, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.243.65@KRBTEST.COM
I20260619 18:47:02.322758 5670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:02.305325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46718 (local address 127.3.243.65:39053)
0619 18:47:02.305506 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:02.305512 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:02.306337 (+ 825us) server_negotiation.cc:407] Connection header received
0619 18:47:02.306574 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:02.306580 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:02.306661 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:02.306766 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:02.308520 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:02.309563 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:02.311358 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:02.311962 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:02.312081 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:02.316580 (+ 4499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:02.316596 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:02.316613 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:02.316660 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:02.320026 (+ 3366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:02.320794 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:02.320803 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:02.320806 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:02.320962 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:02.321282 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:02.321287 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:02.321290 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:02.321721 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:02.322027 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:02.322181 (+ 154us) server_negotiation.cc:299] Negotiation successful
0619 18:47:02.322407 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260619 18:47:02.323450 5373 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46718
I20260619 18:47:02.337765 5751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:02.326253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41114 (local address 127.3.243.126:42779)
0619 18:47:02.326811 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:02.326816 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:02.326829 (+ 13us) server_negotiation.cc:407] Connection header received
0619 18:47:02.326875 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:02.326878 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:02.326934 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:02.327021 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:02.328212 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:02.328924 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:02.330074 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:02.330531 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:02.330636 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:02.332869 (+ 2233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:02.332876 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:02.332879 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:02.332909 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:02.335285 (+ 2376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:02.335883 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:02.335889 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:02.335893 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:02.336028 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:02.336400 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:02.336406 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:02.336409 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:02.336693 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:02.336921 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:02.337229 (+ 308us) server_negotiation.cc:299] Negotiation successful
0619 18:47:02.337452 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":274,"threads_started":1}
W20260619 18:47:02.338356 5172 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41114
I20260619 18:47:02.341657 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 5243
I20260619 18:47:02.353842 5670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:02.342375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46726 (local address 127.3.243.65:39053)
0619 18:47:02.342533 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:02.342538 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:02.342862 (+ 324us) server_negotiation.cc:407] Connection header received
0619 18:47:02.343030 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:02.343034 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:02.343098 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:02.343177 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:02.344288 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:02.344989 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:02.346146 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:02.346549 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:02.346656 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:02.348877 (+ 2221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:02.348882 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:02.348885 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:02.348920 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:02.351338 (+ 2418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:02.351962 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:02.351967 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:02.351971 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:02.352106 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:02.352527 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:02.352533 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:02.352536 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:02.352832 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:02.353028 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:02.353363 (+ 335us) server_negotiation.cc:299] Negotiation successful
0619 18:47:02.353556 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260619 18:47:02.354632 5373 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46726
W20260619 18:47:02.355131 4045 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 5243: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 18:47:02.355332 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 5243
I20260619 18:47:02.381217 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 5382
Jun 19 18:47:02 dist-test-slave-m9wg krb5kdc[5127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894821, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.243.66@KRBTEST.COM
I20260619 18:47:02.398447 5722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:02.382092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51244 (local address 127.3.243.66:36729)
0619 18:47:02.382282 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:02.382288 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:02.382307 (+ 19us) server_negotiation.cc:407] Connection header received
0619 18:47:02.382508 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:02.382513 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:02.382590 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:02.382694 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:02.384318 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:02.385492 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:02.387250 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:02.388009 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:02.388120 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:02.392841 (+ 4721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:02.392852 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:02.392864 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:02.392895 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:02.395774 (+ 2879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:02.396480 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:02.396487 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:02.396489 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:02.396641 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:02.397005 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:02.397008 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:02.397010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:02.397338 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:02.397655 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:02.397900 (+ 245us) server_negotiation.cc:299] Negotiation successful
0619 18:47:02.398102 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260619 18:47:02.399228 5512 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51244
W20260619 18:47:02.399955 4045 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 5382: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 18:47:02.400110 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 5382
I20260619 18:47:02.415721 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 5521
Jun 19 18:47:02 dist-test-slave-m9wg krb5kdc[5127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894821, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.243.67@KRBTEST.COM
I20260619 18:47:02.428803 5716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:02.416431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42114 (local address 127.3.243.67:44041)
0619 18:47:02.416645 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:02.416649 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:02.416667 (+ 18us) server_negotiation.cc:407] Connection header received
0619 18:47:02.416782 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:02.416786 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:02.416848 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:02.416928 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:02.417987 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:02.418656 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:02.419768 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:02.420268 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:02.420361 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:02.423964 (+ 3603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:02.423978 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:02.423995 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:02.424028 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:02.426613 (+ 2585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:02.427114 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:02.427119 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:02.427122 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:02.427238 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:02.427516 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:02.427519 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:02.427521 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:02.427829 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:02.428029 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:02.428338 (+ 309us) server_negotiation.cc:299] Negotiation successful
0619 18:47:02.428548 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260619 18:47:02.429584 5651 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42114
W20260619 18:47:02.430187 4045 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 5521: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 18:47:02.430358 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 5521
I20260619 18:47:02.445327 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 5140
I20260619 18:47:02.446307 5212 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:47:02.558526 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 5140
2026-06-19T18:47:02Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4266 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:47:02 dist-test-slave-m9wg krb5kdc[5756](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:47:02 dist-test-slave-m9wg krb5kdc[5756](info): set up 2 sockets
Jun 19 18:47:02 dist-test-slave-m9wg krb5kdc[5756](info): commencing operation
krb5kdc: starting...
W20260619 18:47:03.552088 4045 mini_kdc.cc:121] Time spent starting KDC: real 0.955s user 0.004s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/krb5kdc/test-user.keytab.
Jun 19 18:47:03 dist-test-slave-m9wg krb5kdc[5756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894823, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:47:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:47:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.243.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab.
Entry for principal kudu/127.3.243.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab.
I20260619 18:47:03.715284 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.126:39867
--webserver_interface=127.3.243.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab
--principal=kudu/127.3.243.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:35697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.126:39867
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:47:04.018625 5769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:47:04.018893 5769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:47:04.019017 5769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:47:04.026086 5769 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:47:04.026189 5769 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 18:47:04.026250 5769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:47:04.026331 5769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:47:04.026392 5769 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:47:04.026494 5769 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:47:04.036259 5769 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:35697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.243.126:39867
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab
--principal=kudu/127.3.243.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.126:39867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.243.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:47:04.037508 5769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:47:04.038784 5769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:47:04.049726 5774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:04.049652 5777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:04.049695 5775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:47:04.050530 5769 server_base.cc:1034] running on GCE node
I20260619 18:47:04.051191 5769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:47:04.052505 5769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:47:04.053704 5769 hybrid_clock.cc:648] HybridClock initialized: now 1781894824053642 us; error 35 us; skew 500 ppm
I20260619 18:47:04.053984 5769 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:47:04 dist-test-slave-m9wg krb5kdc[5756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894824, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:47:04.056857 5769 init.cc:377] Logged in from keytab as kudu/127.3.243.126@KRBTEST.COM (short username kudu)
I20260619 18:47:04.058223 5769 webserver.cc:460] Webserver started at http://127.3.243.126:39685/ using document root <none> and password file <none>
I20260619 18:47:04.058877 5769 fs_manager.cc:359] Metadata directory not provided
I20260619 18:47:04.058967 5769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:47:04.059252 5769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:47:04.062173 5769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/minicluster-data/master-0/data/instance:
uuid: "7647451422aa4ed39c38da3831664f1b"
format_stamp: "Formatted at 2026-06-19 18:47:04 on dist-test-slave-m9wg"
server_key: "e579d20f1e5abeee6201d7e14caba0fc"
server_key_iv: "8909d1e52f94ba18e7f596d1ff39229c"
server_key_version: "encryptionkey@0"
I20260619 18:47:04.062901 5769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/minicluster-data/master-0/wal/instance:
uuid: "7647451422aa4ed39c38da3831664f1b"
format_stamp: "Formatted at 2026-06-19 18:47:04 on dist-test-slave-m9wg"
server_key: "e579d20f1e5abeee6201d7e14caba0fc"
server_key_iv: "8909d1e52f94ba18e7f596d1ff39229c"
server_key_version: "encryptionkey@0"
I20260619 18:47:04.067790 5769 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260619 18:47:04.071084 5784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:04.072388 5769 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 18:47:04.072558 5769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/minicluster-data/master-0/wal
uuid: "7647451422aa4ed39c38da3831664f1b"
format_stamp: "Formatted at 2026-06-19 18:47:04 on dist-test-slave-m9wg"
server_key: "e579d20f1e5abeee6201d7e14caba0fc"
server_key_iv: "8909d1e52f94ba18e7f596d1ff39229c"
server_key_version: "encryptionkey@0"
I20260619 18:47:04.072739 5769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:47:04.086280 5769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:47:04.089990 5769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:47:04.090248 5769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:47:04.109485 5769 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.126:39867
I20260619 18:47:04.109509 5845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.126:39867 every 8 connection(s)
I20260619 18:47:04.110822 5769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
I20260619 18:47:04.115044 5846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:47:04.120867 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 5769
I20260619 18:47:04.121052 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/minicluster-data/master-0/wal/instance
I20260619 18:47:04.121644 4045 external_mini_cluster.cc:1428] Setting key cf53f825347094c4482bfdcb66818ad6
I20260619 18:47:04.125584 5846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7647451422aa4ed39c38da3831664f1b: Bootstrap starting.
I20260619 18:47:04.129159 5846 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7647451422aa4ed39c38da3831664f1b: Neither blocks nor log segments found. Creating new log.
I20260619 18:47:04.130254 5846 log.cc:824] T 00000000000000000000000000000000 P 7647451422aa4ed39c38da3831664f1b: Log is configured to *not* fsync() on all Append() calls
I20260619 18:47:04.133214 5846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7647451422aa4ed39c38da3831664f1b: No bootstrap required, opened a new log
Jun 19 18:47:04 dist-test-slave-m9wg krb5kdc[5756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894823, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.126@KRBTEST.COM
I20260619 18:47:04.142043 5846 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7647451422aa4ed39c38da3831664f1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7647451422aa4ed39c38da3831664f1b" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 39867 } }
I20260619 18:47:04.142524 5846 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7647451422aa4ed39c38da3831664f1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:47:04.142619 5846 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7647451422aa4ed39c38da3831664f1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7647451422aa4ed39c38da3831664f1b, State: Initialized, Role: FOLLOWER
I20260619 18:47:04.143249 5846 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7647451422aa4ed39c38da3831664f1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7647451422aa4ed39c38da3831664f1b" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 39867 } }
I20260619 18:47:04.143433 5849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:04.123499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48392 (local address 127.3.243.126:39867)
0619 18:47:04.124274 (+ 775us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:04.124286 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:04.124325 (+ 39us) server_negotiation.cc:407] Connection header received
0619 18:47:04.125973 (+ 1648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:04.125994 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:04.126382 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:04.126699 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:04.128468 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:04.129744 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:04.131322 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:04.131985 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:04.132109 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:04.136696 (+ 4587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:04.136707 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:04.136721 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:04.136753 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:04.139594 (+ 2841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:04.140388 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:04.140394 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:04.140400 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:04.140561 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:04.140925 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:04.140928 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:04.140930 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:04.141424 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:04.141624 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:04.142044 (+ 420us) server_negotiation.cc:299] Negotiation successful
0619 18:47:04.142349 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":334,"threads_started":1}
I20260619 18:47:04.143512 5846 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7647451422aa4ed39c38da3831664f1b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:47:04.143651 5846 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7647451422aa4ed39c38da3831664f1b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:47:04.143834 5846 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7647451422aa4ed39c38da3831664f1b [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:47:04.146441 5846 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7647451422aa4ed39c38da3831664f1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7647451422aa4ed39c38da3831664f1b" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 39867 } }
I20260619 18:47:04.146899 5846 leader_election.cc:302] T 00000000000000000000000000000000 P 7647451422aa4ed39c38da3831664f1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7647451422aa4ed39c38da3831664f1b; no voters:
I20260619 18:47:04.147368 5846 leader_election.cc:288] T 00000000000000000000000000000000 P 7647451422aa4ed39c38da3831664f1b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:47:04.147500 5851 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7647451422aa4ed39c38da3831664f1b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:47:04.148108 5851 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7647451422aa4ed39c38da3831664f1b [term 1 LEADER]: Becoming Leader. State: Replica: 7647451422aa4ed39c38da3831664f1b, State: Running, Role: LEADER
I20260619 18:47:04.148638 5851 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7647451422aa4ed39c38da3831664f1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7647451422aa4ed39c38da3831664f1b" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 39867 } }
I20260619 18:47:04.149224 5846 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7647451422aa4ed39c38da3831664f1b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:47:04.151189 5853 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7647451422aa4ed39c38da3831664f1b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7647451422aa4ed39c38da3831664f1b. Latest consensus state: current_term: 1 leader_uuid: "7647451422aa4ed39c38da3831664f1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7647451422aa4ed39c38da3831664f1b" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 39867 } } }
I20260619 18:47:04.151595 5853 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7647451422aa4ed39c38da3831664f1b [sys.catalog]: This master's current role is: LEADER
I20260619 18:47:04.151261 5852 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7647451422aa4ed39c38da3831664f1b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7647451422aa4ed39c38da3831664f1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7647451422aa4ed39c38da3831664f1b" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 39867 } } }
I20260619 18:47:04.152055 5852 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7647451422aa4ed39c38da3831664f1b [sys.catalog]: This master's current role is: LEADER
I20260619 18:47:04.156571 5860 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:47:04.160931 5860 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:47:04.169242 5860 catalog_manager.cc:1347] Generated new cluster ID: de5c51642efd460aaf0381487dfe0edf
I20260619 18:47:04.169346 5860 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:47:04.189139 5860 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:47:04.190522 5860 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:47:04.197929 5860 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7647451422aa4ed39c38da3831664f1b: Generated new TSK 0
I20260619 18:47:04.198788 5860 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.243.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.65.keytab.
Entry for principal kudu/127.3.243.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.65.keytab.
I20260619 18:47:04.249483 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.65:0
--local_ip_for_outbound_sockets=127.3.243.65
--webserver_interface=127.3.243.65
--webserver_port=0
--tserver_master_addrs=127.3.243.126:39867
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.65.keytab
--principal=kudu/127.3.243.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:35697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:47:04.556443 5872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:47:04.556718 5872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:47:04.556856 5872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:47:04.564072 5872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:47:04.564203 5872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:47:04.564334 5872 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 18:47:04.564395 5872 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 18:47:04.564502 5872 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.65
I20260619 18:47:04.574133 5872 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:35697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.65.keytab
--principal=kudu/127.3.243.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.243.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.243.126:39867
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.65
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:47:04.575395 5872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:47:04.576810 5872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:47:04.589030 5880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:04.589063 5877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:04.589102 5878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:47:04.589778 5872 server_base.cc:1034] running on GCE node
I20260619 18:47:04.590348 5872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:47:04.591216 5872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:47:04.592407 5872 hybrid_clock.cc:648] HybridClock initialized: now 1781894824592352 us; error 58 us; skew 500 ppm
I20260619 18:47:04.592717 5872 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:47:04 dist-test-slave-m9wg krb5kdc[5756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894824, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:47:04.595675 5872 init.cc:377] Logged in from keytab as kudu/127.3.243.65@KRBTEST.COM (short username kudu)
I20260619 18:47:04.597131 5872 webserver.cc:460] Webserver started at http://127.3.243.65:41799/ using document root <none> and password file <none>
I20260619 18:47:04.597790 5872 fs_manager.cc:359] Metadata directory not provided
I20260619 18:47:04.597904 5872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:47:04.598198 5872 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:47:04.601167 5872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/minicluster-data/ts-0/data/instance:
uuid: "b146718b0a3244aca8d5d949e5d8f13e"
format_stamp: "Formatted at 2026-06-19 18:47:04 on dist-test-slave-m9wg"
server_key: "96b70d47c6ef009cf0e1295cf5193c9c"
server_key_iv: "18512b71b035b7ae460bb0df2a9d89bf"
server_key_version: "encryptionkey@0"
I20260619 18:47:04.601891 5872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/minicluster-data/ts-0/wal/instance:
uuid: "b146718b0a3244aca8d5d949e5d8f13e"
format_stamp: "Formatted at 2026-06-19 18:47:04 on dist-test-slave-m9wg"
server_key: "96b70d47c6ef009cf0e1295cf5193c9c"
server_key_iv: "18512b71b035b7ae460bb0df2a9d89bf"
server_key_version: "encryptionkey@0"
I20260619 18:47:04.606964 5872 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 18:47:04.610299 5887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:04.611534 5872 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260619 18:47:04.611673 5872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/minicluster-data/ts-0/wal
uuid: "b146718b0a3244aca8d5d949e5d8f13e"
format_stamp: "Formatted at 2026-06-19 18:47:04 on dist-test-slave-m9wg"
server_key: "96b70d47c6ef009cf0e1295cf5193c9c"
server_key_iv: "18512b71b035b7ae460bb0df2a9d89bf"
server_key_version: "encryptionkey@0"
I20260619 18:47:04.611850 5872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:47:04.633498 5872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:47:04.638509 5872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:47:04.638832 5872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:47:04.645671 5872 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:47:04.645843 5872 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260619 18:47:04.646026 5872 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:47:04.646184 5872 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 18:47:04 dist-test-slave-m9wg krb5kdc[5756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894824, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.65@KRBTEST.COM for kudu/127.3.243.126@KRBTEST.COM
I20260619 18:47:04.677907 5895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:04.646178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.65:46267 (local address 127.3.243.126:39867)
0619 18:47:04.646643 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:04.646648 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:04.648828 (+ 2180us) server_negotiation.cc:407] Connection header received
0619 18:47:04.650902 (+ 2074us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:04.650909 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:04.651003 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:04.651128 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:04.660391 (+ 9263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:04.661442 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:04.664718 (+ 3276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:04.665316 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:04.665427 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:04.670885 (+ 5458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:04.670891 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:04.670894 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:04.670931 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:04.674230 (+ 3299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:04.675253 (+ 1023us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:04.675259 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:04.675263 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:04.675448 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:04.675864 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:04.675869 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:04.675872 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:04.676287 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:04.676445 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:04.677232 (+ 787us) server_negotiation.cc:299] Negotiation successful
0619 18:47:04.677501 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":235,"threads_started":1}
I20260619 18:47:04.678700 5897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:04.646930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:39867 (local address 127.3.243.65:46267)
0619 18:47:04.647862 (+ 932us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:04.647922 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:04.650564 (+ 2642us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:04.651409 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:04.651421 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:04.652239 (+ 818us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:47:04.653167 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:04.653200 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:04.661714 (+ 8514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:04.661720 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:04.664501 (+ 2781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:04.664525 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:04.665894 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:04.665907 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:04.666285 (+ 378us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:04.667158 (+ 873us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:47:04.667194 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:47:04.670616 (+ 3422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:47:04.674432 (+ 3816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:04.674442 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:04.674474 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:04.675106 (+ 632us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:04.675555 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:04.675560 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:04.675563 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:04.675762 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:04.676446 (+ 684us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:47:04.676463 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:47:04.676857 (+ 394us) client_negotiation.cc:769] Sending connection context
0619 18:47:04.677109 (+ 252us) client_negotiation.cc:240] Negotiation successful
0619 18:47:04.677575 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"thread_start_us":404,"threads_started":1}
I20260619 18:47:04.688267 5872 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.65:45537
I20260619 18:47:04.688455 6005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.65:45537 every 8 connection(s)
I20260619 18:47:04.689726 5872 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/minicluster-data/ts-0/data/info.pb
I20260619 18:47:04.692047 6007 heartbeater.cc:347] Connected to a master server at 127.3.243.126:39867
I20260619 18:47:04.692312 6007 heartbeater.cc:464] Registering TS with master...
I20260619 18:47:04.693207 6007 heartbeater.cc:511] Master 127.3.243.126:39867 requested a full tablet report, sending...
I20260619 18:47:04.695765 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 5872
I20260619 18:47:04.695986 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/minicluster-data/ts-0/wal/instance
I20260619 18:47:04.696851 4045 external_mini_cluster.cc:1428] Setting key bc9d276decc52ab6dacb0376df3316b6
I20260619 18:47:04.697690 5801 ts_manager.cc:194] Registered new tserver with Master: b146718b0a3244aca8d5d949e5d8f13e (127.3.243.65:45537)
I20260619 18:47:04.699795 5801 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.65@KRBTEST.COM'} at 127.3.243.65:46267
I20260619 18:47:04.705909 5895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:04.690354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.65:44529 (local address 127.3.243.126:39867)
0619 18:47:04.690546 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:04.690553 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:04.691659 (+ 1106us) server_negotiation.cc:407] Connection header received
0619 18:47:04.691815 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:04.691819 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:04.691901 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:04.691993 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:04.693255 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:04.694415 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:04.695796 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:04.696512 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:04.696635 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:04.699075 (+ 2440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:04.699082 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:04.699086 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:04.699120 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:04.703340 (+ 4220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:04.703978 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:04.703983 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:04.703986 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:04.704126 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:04.704530 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:04.704534 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:04.704537 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:04.704855 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:04.704985 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:04.705370 (+ 385us) server_negotiation.cc:299] Negotiation successful
0619 18:47:04.705586 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 18:47:04.706020 6009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:04.690393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:39867 (local address 127.3.243.65:44529)
0619 18:47:04.691553 (+ 1160us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:04.691573 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:04.691706 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:04.692053 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:04.692057 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:04.692719 (+ 662us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:47:04.693067 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:04.693081 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:04.694550 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:04.694554 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:04.695637 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:04.695653 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:04.696748 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:04.696755 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:04.696976 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:04.697936 (+ 960us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:47:04.697960 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:47:04.698883 (+ 923us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:47:04.703466 (+ 4583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:04.703470 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:04.703473 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:04.703878 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:04.704248 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:04.704252 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:04.704256 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:04.704407 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:04.704982 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:47:04.704986 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:47:04.705154 (+ 168us) client_negotiation.cc:769] Sending connection context
0619 18:47:04.705270 (+ 116us) client_negotiation.cc:240] Negotiation successful
0619 18:47:04.705575 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1028,"thread_start_us":904,"threads_started":1}
WARNING: no policy specified for kudu/127.3.243.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.66.keytab.
Entry for principal kudu/127.3.243.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.66.keytab.
I20260619 18:47:04.734838 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.66:0
--local_ip_for_outbound_sockets=127.3.243.66
--webserver_interface=127.3.243.66
--webserver_port=0
--tserver_master_addrs=127.3.243.126:39867
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.66.keytab
--principal=kudu/127.3.243.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:35697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:47:05.040593 6012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:47:05.040863 6012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:47:05.041005 6012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:47:05.048323 6012 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:47:05.048425 6012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:47:05.048542 6012 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 18:47:05.048602 6012 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 18:47:05.048698 6012 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.66
I20260619 18:47:05.058409 6012 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:35697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.66.keytab
--principal=kudu/127.3.243.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.243.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.243.126:39867
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.66
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:47:05.059648 6012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:47:05.060985 6012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:47:05.073220 6018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:05.073238 6020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:05.073303 6017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:47:05.074014 6012 server_base.cc:1034] running on GCE node
I20260619 18:47:05.074604 6012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:47:05.075481 6012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:47:05.076674 6012 hybrid_clock.cc:648] HybridClock initialized: now 1781894825076619 us; error 49 us; skew 500 ppm
I20260619 18:47:05.077018 6012 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:47:05 dist-test-slave-m9wg krb5kdc[5756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894825, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:47:05.079931 6012 init.cc:377] Logged in from keytab as kudu/127.3.243.66@KRBTEST.COM (short username kudu)
I20260619 18:47:05.081310 6012 webserver.cc:460] Webserver started at http://127.3.243.66:43967/ using document root <none> and password file <none>
I20260619 18:47:05.081946 6012 fs_manager.cc:359] Metadata directory not provided
I20260619 18:47:05.082056 6012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:47:05.082337 6012 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:47:05.085184 6012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/minicluster-data/ts-1/data/instance:
uuid: "e3dbb92deec94038b08d065aa47a1bf1"
format_stamp: "Formatted at 2026-06-19 18:47:05 on dist-test-slave-m9wg"
server_key: "398b4b78ebc88d351911ffffe9fa083c"
server_key_iv: "abcc927b9ac267ad046d5cd6ab8be1fa"
server_key_version: "encryptionkey@0"
I20260619 18:47:05.085877 6012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/minicluster-data/ts-1/wal/instance:
uuid: "e3dbb92deec94038b08d065aa47a1bf1"
format_stamp: "Formatted at 2026-06-19 18:47:05 on dist-test-slave-m9wg"
server_key: "398b4b78ebc88d351911ffffe9fa083c"
server_key_iv: "abcc927b9ac267ad046d5cd6ab8be1fa"
server_key_version: "encryptionkey@0"
I20260619 18:47:05.090685 6012 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 18:47:05.094008 6027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:05.095300 6012 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 18:47:05.095438 6012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/minicluster-data/ts-1/wal
uuid: "e3dbb92deec94038b08d065aa47a1bf1"
format_stamp: "Formatted at 2026-06-19 18:47:05 on dist-test-slave-m9wg"
server_key: "398b4b78ebc88d351911ffffe9fa083c"
server_key_iv: "abcc927b9ac267ad046d5cd6ab8be1fa"
server_key_version: "encryptionkey@0"
I20260619 18:47:05.095619 6012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:47:05.111694 6012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:47:05.116480 6012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:47:05.116779 6012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:47:05.119995 6012 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:47:05.120112 6012 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:05.120253 6012 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:47:05.120335 6012 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 18:47:05 dist-test-slave-m9wg krb5kdc[5756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894825, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.66@KRBTEST.COM for kudu/127.3.243.126@KRBTEST.COM
I20260619 18:47:05.143455 5895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:05.120640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.66:52147 (local address 127.3.243.126:39867)
0619 18:47:05.120881 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:05.120888 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:05.122874 (+ 1986us) server_negotiation.cc:407] Connection header received
0619 18:47:05.125683 (+ 2809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:05.125689 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:05.125805 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:05.125936 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:05.128001 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:05.129064 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:05.131531 (+ 2467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:05.132087 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:05.132253 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:05.137151 (+ 4898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:05.137158 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:05.137164 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:05.137208 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:05.140427 (+ 3219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:05.141137 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:05.141143 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:05.141147 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:05.141278 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:05.141716 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:05.141721 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:05.141724 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:05.142010 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:05.142145 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:05.142863 (+ 718us) server_negotiation.cc:299] Negotiation successful
0619 18:47:05.143112 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 18:47:05.144752 6036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:05.121812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:39867 (local address 127.3.243.66:52147)
0619 18:47:05.122716 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:05.122768 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:05.125357 (+ 2589us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:05.126207 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:05.126221 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:05.126943 (+ 722us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:47:05.127774 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:05.127803 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:05.129234 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:05.129240 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:05.131322 (+ 2082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:05.131337 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:05.132559 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:05.132567 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:05.132839 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:05.133540 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:47:05.133579 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:47:05.136888 (+ 3309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:47:05.140561 (+ 3673us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:05.140571 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:05.140589 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:05.141023 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:05.141392 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:05.141396 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:05.141398 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:05.141605 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:05.142150 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:47:05.142158 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:47:05.142522 (+ 364us) client_negotiation.cc:769] Sending connection context
0619 18:47:05.142762 (+ 240us) client_negotiation.cc:240] Negotiation successful
0619 18:47:05.143291 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":622,"thread_start_us":382,"threads_started":1}
I20260619 18:47:05.162925 5895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:05.150874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.66:42823 (local address 127.3.243.126:39867)
0619 18:47:05.151067 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:05.151074 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:05.151543 (+ 469us) server_negotiation.cc:407] Connection header received
0619 18:47:05.151686 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:05.151692 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:05.151774 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:05.151883 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:05.152963 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:05.153928 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:05.155217 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:05.155629 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:05.155735 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:05.157515 (+ 1780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:05.157520 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:05.157523 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:05.157555 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:05.159841 (+ 2286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:05.160575 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:05.160580 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:05.160583 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:05.160745 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:05.161195 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:05.161199 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:05.161201 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:05.161618 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:05.161742 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:05.162344 (+ 602us) server_negotiation.cc:299] Negotiation successful
0619 18:47:05.162596 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 18:47:05.163344 6104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:05.150871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:39867 (local address 127.3.243.66:42823)
0619 18:47:05.151457 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:05.151475 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:05.151587 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:05.151893 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:05.151896 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:05.152354 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:47:05.152750 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:05.152767 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:05.154070 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:05.154073 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:05.155077 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:05.155089 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:05.155800 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:05.155806 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:05.155958 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:05.156561 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:47:05.156577 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:47:05.157329 (+ 752us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:47:05.159972 (+ 2643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:05.159978 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:05.159982 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:05.160469 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:05.160851 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:05.160856 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:05.160859 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:05.161046 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:05.161816 (+ 770us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:47:05.161821 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:47:05.162057 (+ 236us) client_negotiation.cc:769] Sending connection context
0619 18:47:05.162211 (+ 154us) client_negotiation.cc:240] Negotiation successful
0619 18:47:05.162663 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":296,"threads_started":1}
I20260619 18:47:05.172423 6012 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.66:35763
I20260619 18:47:05.172451 6146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.66:35763 every 8 connection(s)
I20260619 18:47:05.173600 6012 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/minicluster-data/ts-1/data/info.pb
I20260619 18:47:05.175493 6147 heartbeater.cc:347] Connected to a master server at 127.3.243.126:39867
I20260619 18:47:05.175699 6147 heartbeater.cc:464] Registering TS with master...
I20260619 18:47:05.176385 6147 heartbeater.cc:511] Master 127.3.243.126:39867 requested a full tablet report, sending...
I20260619 18:47:05.177621 5801 ts_manager.cc:194] Registered new tserver with Master: e3dbb92deec94038b08d065aa47a1bf1 (127.3.243.66:35763)
I20260619 18:47:05.178411 5801 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.66@KRBTEST.COM'} at 127.3.243.66:52147
I20260619 18:47:05.181293 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 6012
I20260619 18:47:05.181437 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/minicluster-data/ts-1/wal/instance
I20260619 18:47:05.181880 4045 external_mini_cluster.cc:1428] Setting key 13a16152c1e2a71f333bd5d5c3d02216
WARNING: no policy specified for kudu/127.3.243.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.67.keytab.
Entry for principal kudu/127.3.243.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.67.keytab.
I20260619 18:47:05.216779 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.67:0
--local_ip_for_outbound_sockets=127.3.243.67
--webserver_interface=127.3.243.67
--webserver_port=0
--tserver_master_addrs=127.3.243.126:39867
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.67.keytab
--principal=kudu/127.3.243.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:35697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:47:05.519414 6151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:47:05.519693 6151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:47:05.519830 6151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:47:05.526804 6151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:47:05.526921 6151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:47:05.527043 6151 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 18:47:05.527108 6151 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 18:47:05.527210 6151 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.67
I20260619 18:47:05.536650 6151 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:35697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.67.keytab
--principal=kudu/127.3.243.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.243.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.243.126:39867
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.67
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:47:05.537870 6151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:47:05.539163 6151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:47:05.551286 6156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:05.551241 6159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:05.551324 6157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:47:05.552452 6151 server_base.cc:1034] running on GCE node
I20260619 18:47:05.553053 6151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:47:05.553972 6151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:47:05.555181 6151 hybrid_clock.cc:648] HybridClock initialized: now 1781894825555119 us; error 53 us; skew 500 ppm
I20260619 18:47:05.555496 6151 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:47:05 dist-test-slave-m9wg krb5kdc[5756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894825, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:47:05.558660 6151 init.cc:377] Logged in from keytab as kudu/127.3.243.67@KRBTEST.COM (short username kudu)
I20260619 18:47:05.560057 6151 webserver.cc:460] Webserver started at http://127.3.243.67:36479/ using document root <none> and password file <none>
I20260619 18:47:05.560796 6151 fs_manager.cc:359] Metadata directory not provided
I20260619 18:47:05.560897 6151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:47:05.561156 6151 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:47:05.564034 6151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/minicluster-data/ts-2/data/instance:
uuid: "efb278fc23954c9d86d8c54f68b6462e"
format_stamp: "Formatted at 2026-06-19 18:47:05 on dist-test-slave-m9wg"
server_key: "28814973ecc89e4727c990b6d5c8a3e2"
server_key_iv: "d85ffbead8bf76fbe7ed0369d3916ac1"
server_key_version: "encryptionkey@0"
I20260619 18:47:05.564816 6151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/minicluster-data/ts-2/wal/instance:
uuid: "efb278fc23954c9d86d8c54f68b6462e"
format_stamp: "Formatted at 2026-06-19 18:47:05 on dist-test-slave-m9wg"
server_key: "28814973ecc89e4727c990b6d5c8a3e2"
server_key_iv: "d85ffbead8bf76fbe7ed0369d3916ac1"
server_key_version: "encryptionkey@0"
I20260619 18:47:05.569949 6151 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 18:47:05.573356 6166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:05.574684 6151 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 18:47:05.574839 6151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/minicluster-data/ts-2/wal
uuid: "efb278fc23954c9d86d8c54f68b6462e"
format_stamp: "Formatted at 2026-06-19 18:47:05 on dist-test-slave-m9wg"
server_key: "28814973ecc89e4727c990b6d5c8a3e2"
server_key_iv: "d85ffbead8bf76fbe7ed0369d3916ac1"
server_key_version: "encryptionkey@0"
I20260619 18:47:05.575032 6151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:47:05.597244 6151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:47:05.601672 6151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:47:05.601907 6151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:47:05.604501 6151 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:47:05.604607 6151 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:05.604714 6151 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:47:05.604786 6151 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 18:47:05 dist-test-slave-m9wg krb5kdc[5756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894825, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.67@KRBTEST.COM for kudu/127.3.243.126@KRBTEST.COM
I20260619 18:47:05.626893 5895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:05.605176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.67:37069 (local address 127.3.243.126:39867)
0619 18:47:05.605367 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:05.605374 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:05.606764 (+ 1390us) server_negotiation.cc:407] Connection header received
0619 18:47:05.609017 (+ 2253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:05.609025 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:05.609124 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:05.609242 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:05.611326 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:05.612349 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:05.614759 (+ 2410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:05.615368 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:05.615493 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:05.620332 (+ 4839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:05.620337 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:05.620340 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:05.620373 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:05.623543 (+ 3170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:05.624564 (+ 1021us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:05.624569 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:05.624573 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:05.624705 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:05.625159 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:05.625164 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:05.625168 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:05.625450 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:05.625575 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:05.626303 (+ 728us) server_negotiation.cc:299] Negotiation successful
0619 18:47:05.626542 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 18:47:05.628096 6175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:05.605741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:39867 (local address 127.3.243.67:37069)
0619 18:47:05.606576 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:05.606629 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:05.608656 (+ 2027us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:05.609507 (+ 851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:05.609516 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:05.610238 (+ 722us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:47:05.611118 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:05.611155 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:05.612583 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:05.612588 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:05.614610 (+ 2022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:05.614623 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:05.615629 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:05.615637 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:05.615891 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:05.616852 (+ 961us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:47:05.616884 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:47:05.620020 (+ 3136us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:47:05.623677 (+ 3657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:05.623686 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:05.623709 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:05.624379 (+ 670us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:05.624808 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:05.624813 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:05.624817 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:05.625043 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:05.625568 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:47:05.625576 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:47:05.625922 (+ 346us) client_negotiation.cc:769] Sending connection context
0619 18:47:05.626154 (+ 232us) client_negotiation.cc:240] Negotiation successful
0619 18:47:05.626655 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":345,"threads_started":1}
I20260619 18:47:05.647754 5895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:05.633770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.67:59211 (local address 127.3.243.126:39867)
0619 18:47:05.633916 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:05.633920 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:05.634514 (+ 594us) server_negotiation.cc:407] Connection header received
0619 18:47:05.634688 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:05.634693 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:05.634758 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:05.634833 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:05.635750 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:05.636644 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:05.638684 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:05.639202 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:05.639321 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:05.641781 (+ 2460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:05.641787 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:05.641791 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:05.641831 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:05.645111 (+ 3280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:05.645869 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:05.645875 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:05.645878 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:05.646011 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:05.646339 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:05.646345 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:05.646348 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:05.646630 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:05.646761 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:05.647157 (+ 396us) server_negotiation.cc:299] Negotiation successful
0619 18:47:05.647375 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 18:47:05.647850 6251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:05.633849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:39867 (local address 127.3.243.67:59211)
0619 18:47:05.634408 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:05.634423 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:05.634567 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:05.634870 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:05.634874 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:05.635325 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:47:05.635613 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:05.635622 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:05.636803 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:05.636809 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:05.638515 (+ 1706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:05.638532 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:05.639549 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:05.639555 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:05.639769 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:05.640439 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:47:05.640462 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:47:05.641553 (+ 1091us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:47:05.645278 (+ 3725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:05.645283 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:05.645287 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:05.645758 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:05.646113 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:05.646118 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:05.646122 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:05.646240 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:05.646740 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:47:05.646745 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:47:05.646924 (+ 179us) client_negotiation.cc:769] Sending connection context
0619 18:47:05.647060 (+ 136us) client_negotiation.cc:240] Negotiation successful
0619 18:47:05.647383 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":302,"threads_started":1}
I20260619 18:47:05.654435 6151 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.67:35465
I20260619 18:47:05.654471 6285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.67:35465 every 8 connection(s)
I20260619 18:47:05.655622 6151 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/minicluster-data/ts-2/data/info.pb
I20260619 18:47:05.657548 6286 heartbeater.cc:347] Connected to a master server at 127.3.243.126:39867
I20260619 18:47:05.657750 6286 heartbeater.cc:464] Registering TS with master...
I20260619 18:47:05.658339 6286 heartbeater.cc:511] Master 127.3.243.126:39867 requested a full tablet report, sending...
I20260619 18:47:05.659585 5801 ts_manager.cc:194] Registered new tserver with Master: efb278fc23954c9d86d8c54f68b6462e (127.3.243.67:35465)
I20260619 18:47:05.660449 5801 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.67@KRBTEST.COM'} at 127.3.243.67:37069
I20260619 18:47:05.662218 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 6151
I20260619 18:47:05.662366 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/minicluster-data/ts-2/wal/instance
I20260619 18:47:05.662812 4045 external_mini_cluster.cc:1428] Setting key 02ab6359c6e2b46d0de3ba9cffe289c8
I20260619 18:47:05.665457 4045 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 18:47:05 dist-test-slave-m9wg krb5kdc[5756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894825, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 18:47:05 dist-test-slave-m9wg krb5kdc[5756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894825, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.126@KRBTEST.COM
I20260619 18:47:05.706720 6007 heartbeater.cc:503] Master 127.3.243.126:39867 was elected leader, sending a full tablet report...
I20260619 18:47:05.711066 5895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:05.698080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48404 (local address 127.3.243.126:39867)
0619 18:47:05.698295 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:05.698301 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:05.698591 (+ 290us) server_negotiation.cc:407] Connection header received
0619 18:47:05.698765 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:05.698770 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:05.698845 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:05.698941 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:05.700331 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:05.701095 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:05.702318 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:05.702726 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:05.702819 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:05.706414 (+ 3595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:05.706419 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:05.706422 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:05.706455 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:05.708995 (+ 2540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:05.709548 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:05.709551 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:05.709554 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:05.709667 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:05.709938 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:05.709941 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:05.709943 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:05.710182 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:05.710353 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:05.710564 (+ 211us) server_negotiation.cc:299] Negotiation successful
0619 18:47:05.710722 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 18:47:05.714843 5801 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48404:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:47:05.721442 5801 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:47:05.755021 6299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:05.733051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.65:45537 (local address 127.0.0.1:50528)
0619 18:47:05.740113 (+ 7062us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:05.740187 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:05.740398 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:05.744441 (+ 4043us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:05.744461 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:05.744501 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:47:05.744914 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:05.744926 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:05.748024 (+ 3098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:05.748032 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:05.750732 (+ 2700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:05.750753 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:05.754020 (+ 3267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:05.754025 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:05.754220 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:05.754273 (+ 53us) client_negotiation.cc:769] Sending connection context
0619 18:47:05.754407 (+ 134us) client_negotiation.cc:240] Negotiation successful
0619 18:47:05.754624 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6745,"thread_start_us":6584,"threads_started":1}
I20260619 18:47:05.755497 6300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:05.733190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.66:35763 (local address 127.0.0.1:32794)
0619 18:47:05.741677 (+ 8487us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:05.741693 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:05.741801 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:05.747880 (+ 6079us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:05.747885 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:05.747915 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:47:05.748471 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:05.748483 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:05.750622 (+ 2139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:05.750628 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:05.752858 (+ 2230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:05.752872 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:05.754582 (+ 1710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:05.754589 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:05.754765 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:05.754900 (+ 135us) client_negotiation.cc:769] Sending connection context
0619 18:47:05.754984 (+ 84us) client_negotiation.cc:240] Negotiation successful
0619 18:47:05.755116 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8378,"mutex_wait_us":55,"thread_start_us":8237,"threads_started":1}
I20260619 18:47:05.755571 6302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:05.733750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32794 (local address 127.3.243.66:35763)
0619 18:47:05.746991 (+ 13241us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:05.747001 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:05.747035 (+ 34us) server_negotiation.cc:407] Connection header received
0619 18:47:05.747140 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:05.747149 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:05.747662 (+ 513us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:05.748716 (+ 1054us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:47:05.748922 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:05.750454 (+ 1532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:05.753063 (+ 2609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:05.754345 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:05.754851 (+ 506us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:05.755042 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:05.755134 (+ 92us) server_negotiation.cc:299] Negotiation successful
0619 18:47:05.755294 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13052,"thread_start_us":13459,"threads_started":1}
I20260619 18:47:05.755735 6301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:05.733267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50528 (local address 127.3.243.65:45537)
0619 18:47:05.742275 (+ 9008us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:05.742286 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:05.742320 (+ 34us) server_negotiation.cc:407] Connection header received
0619 18:47:05.742430 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:05.742439 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:05.742749 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:05.742959 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:47:05.745149 (+ 2190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:05.746701 (+ 1552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:05.750956 (+ 4255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:05.753715 (+ 2759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:05.753839 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:05.754054 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:05.755312 (+ 1258us) server_negotiation.cc:299] Negotiation successful
0619 18:47:05.755473 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8825,"thread_start_us":14420,"threads_started":1}
I20260619 18:47:05.759549 6077 tablet_service.cc:1467] Processing CreateTablet for tablet 50b6ac7278c84239bb7ec4c97ee2eb78 (DEFAULT_TABLE table=test-table [id=d80636f301c64fc4953890c02155a21e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:47:05.759549 5938 tablet_service.cc:1467] Processing CreateTablet for tablet 50b6ac7278c84239bb7ec4c97ee2eb78 (DEFAULT_TABLE table=test-table [id=d80636f301c64fc4953890c02155a21e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:47:05.761063 5938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50b6ac7278c84239bb7ec4c97ee2eb78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:47:05.761077 6077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50b6ac7278c84239bb7ec4c97ee2eb78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:47:05.761435 6303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:05.742044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.67:35465 (local address 127.0.0.1:39980)
0619 18:47:05.753668 (+ 11624us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:05.753687 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:05.753810 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:05.754406 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:05.754411 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:05.754435 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:47:05.754888 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:05.754900 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:05.758016 (+ 3116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:05.758019 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:05.759277 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:05.759289 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:05.760641 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:05.760647 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:05.760816 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:05.760843 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 18:47:05.760920 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 18:47:05.761068 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11511,"mutex_wait_us":32,"thread_start_us":11385,"threads_started":1}
I20260619 18:47:05.761435 6304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:05.742174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39980 (local address 127.3.243.67:35465)
0619 18:47:05.753136 (+ 10962us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:05.753145 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:05.753800 (+ 655us) server_negotiation.cc:407] Connection header received
0619 18:47:05.753963 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:05.753972 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:05.754225 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:05.754399 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:47:05.755738 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:05.757872 (+ 2134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:05.759430 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:05.760454 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:05.760564 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:05.760785 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:05.760974 (+ 189us) server_negotiation.cc:299] Negotiation successful
0619 18:47:05.761136 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10780,"thread_start_us":10954,"threads_started":1}
I20260619 18:47:05.764565 6216 tablet_service.cc:1467] Processing CreateTablet for tablet 50b6ac7278c84239bb7ec4c97ee2eb78 (DEFAULT_TABLE table=test-table [id=d80636f301c64fc4953890c02155a21e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:47:05.765882 6216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50b6ac7278c84239bb7ec4c97ee2eb78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:47:05.775411 6306 tablet_bootstrap.cc:492] T 50b6ac7278c84239bb7ec4c97ee2eb78 P e3dbb92deec94038b08d065aa47a1bf1: Bootstrap starting.
I20260619 18:47:05.777108 6305 tablet_bootstrap.cc:492] T 50b6ac7278c84239bb7ec4c97ee2eb78 P b146718b0a3244aca8d5d949e5d8f13e: Bootstrap starting.
I20260619 18:47:05.779222 6306 tablet_bootstrap.cc:654] T 50b6ac7278c84239bb7ec4c97ee2eb78 P e3dbb92deec94038b08d065aa47a1bf1: Neither blocks nor log segments found. Creating new log.
I20260619 18:47:05.780432 6306 log.cc:824] T 50b6ac7278c84239bb7ec4c97ee2eb78 P e3dbb92deec94038b08d065aa47a1bf1: Log is configured to *not* fsync() on all Append() calls
I20260619 18:47:05.780849 6307 tablet_bootstrap.cc:492] T 50b6ac7278c84239bb7ec4c97ee2eb78 P efb278fc23954c9d86d8c54f68b6462e: Bootstrap starting.
I20260619 18:47:05.782286 6305 tablet_bootstrap.cc:654] T 50b6ac7278c84239bb7ec4c97ee2eb78 P b146718b0a3244aca8d5d949e5d8f13e: Neither blocks nor log segments found. Creating new log.
I20260619 18:47:05.783299 6305 log.cc:824] T 50b6ac7278c84239bb7ec4c97ee2eb78 P b146718b0a3244aca8d5d949e5d8f13e: Log is configured to *not* fsync() on all Append() calls
I20260619 18:47:05.784265 6307 tablet_bootstrap.cc:654] T 50b6ac7278c84239bb7ec4c97ee2eb78 P efb278fc23954c9d86d8c54f68b6462e: Neither blocks nor log segments found. Creating new log.
I20260619 18:47:05.784449 6306 tablet_bootstrap.cc:492] T 50b6ac7278c84239bb7ec4c97ee2eb78 P e3dbb92deec94038b08d065aa47a1bf1: No bootstrap required, opened a new log
I20260619 18:47:05.784734 6306 ts_tablet_manager.cc:1397] T 50b6ac7278c84239bb7ec4c97ee2eb78 P e3dbb92deec94038b08d065aa47a1bf1: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.005s
I20260619 18:47:05.785444 6307 log.cc:824] T 50b6ac7278c84239bb7ec4c97ee2eb78 P efb278fc23954c9d86d8c54f68b6462e: Log is configured to *not* fsync() on all Append() calls
I20260619 18:47:05.785846 6305 tablet_bootstrap.cc:492] T 50b6ac7278c84239bb7ec4c97ee2eb78 P b146718b0a3244aca8d5d949e5d8f13e: No bootstrap required, opened a new log
I20260619 18:47:05.786088 6305 ts_tablet_manager.cc:1397] T 50b6ac7278c84239bb7ec4c97ee2eb78 P b146718b0a3244aca8d5d949e5d8f13e: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260619 18:47:05.790638 6307 tablet_bootstrap.cc:492] T 50b6ac7278c84239bb7ec4c97ee2eb78 P efb278fc23954c9d86d8c54f68b6462e: No bootstrap required, opened a new log
I20260619 18:47:05.790900 6307 ts_tablet_manager.cc:1397] T 50b6ac7278c84239bb7ec4c97ee2eb78 P efb278fc23954c9d86d8c54f68b6462e: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260619 18:47:05.793880 6305 raft_consensus.cc:348] T 50b6ac7278c84239bb7ec4c97ee2eb78 P b146718b0a3244aca8d5d949e5d8f13e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b146718b0a3244aca8d5d949e5d8f13e" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 45537 } } peers { permanent_uuid: "e3dbb92deec94038b08d065aa47a1bf1" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 35763 } } peers { permanent_uuid: "efb278fc23954c9d86d8c54f68b6462e" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 35465 } }
I20260619 18:47:05.794247 6305 raft_consensus.cc:374] T 50b6ac7278c84239bb7ec4c97ee2eb78 P b146718b0a3244aca8d5d949e5d8f13e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:47:05.794366 6305 raft_consensus.cc:729] T 50b6ac7278c84239bb7ec4c97ee2eb78 P b146718b0a3244aca8d5d949e5d8f13e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b146718b0a3244aca8d5d949e5d8f13e, State: Initialized, Role: FOLLOWER
I20260619 18:47:05.794771 6306 raft_consensus.cc:348] T 50b6ac7278c84239bb7ec4c97ee2eb78 P e3dbb92deec94038b08d065aa47a1bf1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b146718b0a3244aca8d5d949e5d8f13e" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 45537 } } peers { permanent_uuid: "e3dbb92deec94038b08d065aa47a1bf1" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 35763 } } peers { permanent_uuid: "efb278fc23954c9d86d8c54f68b6462e" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 35465 } }
I20260619 18:47:05.794960 6305 consensus_queue.cc:260] T 50b6ac7278c84239bb7ec4c97ee2eb78 P b146718b0a3244aca8d5d949e5d8f13e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b146718b0a3244aca8d5d949e5d8f13e" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 45537 } } peers { permanent_uuid: "e3dbb92deec94038b08d065aa47a1bf1" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 35763 } } peers { permanent_uuid: "efb278fc23954c9d86d8c54f68b6462e" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 35465 } }
I20260619 18:47:05.795171 6306 raft_consensus.cc:374] T 50b6ac7278c84239bb7ec4c97ee2eb78 P e3dbb92deec94038b08d065aa47a1bf1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:47:05.795313 6306 raft_consensus.cc:729] T 50b6ac7278c84239bb7ec4c97ee2eb78 P e3dbb92deec94038b08d065aa47a1bf1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3dbb92deec94038b08d065aa47a1bf1, State: Initialized, Role: FOLLOWER
I20260619 18:47:05.795938 6306 consensus_queue.cc:260] T 50b6ac7278c84239bb7ec4c97ee2eb78 P e3dbb92deec94038b08d065aa47a1bf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b146718b0a3244aca8d5d949e5d8f13e" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 45537 } } peers { permanent_uuid: "e3dbb92deec94038b08d065aa47a1bf1" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 35763 } } peers { permanent_uuid: "efb278fc23954c9d86d8c54f68b6462e" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 35465 } }
I20260619 18:47:05.800686 6305 ts_tablet_manager.cc:1428] T 50b6ac7278c84239bb7ec4c97ee2eb78 P b146718b0a3244aca8d5d949e5d8f13e: Time spent starting tablet: real 0.015s user 0.009s sys 0.004s
I20260619 18:47:05.801575 6147 heartbeater.cc:503] Master 127.3.243.126:39867 was elected leader, sending a full tablet report...
I20260619 18:47:05.801568 6307 raft_consensus.cc:348] T 50b6ac7278c84239bb7ec4c97ee2eb78 P efb278fc23954c9d86d8c54f68b6462e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b146718b0a3244aca8d5d949e5d8f13e" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 45537 } } peers { permanent_uuid: "e3dbb92deec94038b08d065aa47a1bf1" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 35763 } } peers { permanent_uuid: "efb278fc23954c9d86d8c54f68b6462e" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 35465 } }
I20260619 18:47:05.801975 6307 raft_consensus.cc:374] T 50b6ac7278c84239bb7ec4c97ee2eb78 P efb278fc23954c9d86d8c54f68b6462e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:47:05.802067 6307 raft_consensus.cc:729] T 50b6ac7278c84239bb7ec4c97ee2eb78 P efb278fc23954c9d86d8c54f68b6462e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efb278fc23954c9d86d8c54f68b6462e, State: Initialized, Role: FOLLOWER
I20260619 18:47:05.802291 6306 ts_tablet_manager.cc:1428] T 50b6ac7278c84239bb7ec4c97ee2eb78 P e3dbb92deec94038b08d065aa47a1bf1: Time spent starting tablet: real 0.017s user 0.010s sys 0.007s
I20260619 18:47:05.802621 6307 consensus_queue.cc:260] T 50b6ac7278c84239bb7ec4c97ee2eb78 P efb278fc23954c9d86d8c54f68b6462e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b146718b0a3244aca8d5d949e5d8f13e" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 45537 } } peers { permanent_uuid: "e3dbb92deec94038b08d065aa47a1bf1" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 35763 } } peers { permanent_uuid: "efb278fc23954c9d86d8c54f68b6462e" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 35465 } }
I20260619 18:47:05.803606 6286 heartbeater.cc:503] Master 127.3.243.126:39867 was elected leader, sending a full tablet report...
I20260619 18:47:05.804126 6307 ts_tablet_manager.cc:1428] T 50b6ac7278c84239bb7ec4c97ee2eb78 P efb278fc23954c9d86d8c54f68b6462e: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
W20260619 18:47:05.907130 6287 tablet.cc:2369] T 50b6ac7278c84239bb7ec4c97ee2eb78 P efb278fc23954c9d86d8c54f68b6462e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 18:47:05.925468 6148 tablet.cc:2369] T 50b6ac7278c84239bb7ec4c97ee2eb78 P e3dbb92deec94038b08d065aa47a1bf1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 18:47:05.942215 6008 tablet.cc:2369] T 50b6ac7278c84239bb7ec4c97ee2eb78 P b146718b0a3244aca8d5d949e5d8f13e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:47:06.022531 6313 raft_consensus.cc:482] T 50b6ac7278c84239bb7ec4c97ee2eb78 P efb278fc23954c9d86d8c54f68b6462e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:47:06.022862 6313 raft_consensus.cc:504] T 50b6ac7278c84239bb7ec4c97ee2eb78 P efb278fc23954c9d86d8c54f68b6462e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b146718b0a3244aca8d5d949e5d8f13e" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 45537 } } peers { permanent_uuid: "e3dbb92deec94038b08d065aa47a1bf1" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 35763 } } peers { permanent_uuid: "efb278fc23954c9d86d8c54f68b6462e" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 35465 } }
I20260619 18:47:06.024617 6313 leader_election.cc:288] T 50b6ac7278c84239bb7ec4c97ee2eb78 P efb278fc23954c9d86d8c54f68b6462e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b146718b0a3244aca8d5d949e5d8f13e (127.3.243.65:45537), e3dbb92deec94038b08d065aa47a1bf1 (127.3.243.66:35763)
I20260619 18:47:06.030138 6175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:06.024981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.66:35763 (local address 127.3.243.67:51447)
0619 18:47:06.025174 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:06.025193 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:06.025326 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:06.025926 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:06.025931 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:06.025960 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:47:06.026110 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:06.026117 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:06.027063 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:06.027067 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:06.028374 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:06.028386 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:06.029411 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:06.029417 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:06.029603 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:06.029635 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 18:47:06.029713 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 18:47:06.029864 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260619 18:47:06.030138 6302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:06.025398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.67:51447 (local address 127.3.243.66:35763)
0619 18:47:06.025607 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:06.025611 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:06.025626 (+ 15us) server_negotiation.cc:407] Connection header received
0619 18:47:06.025686 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:06.025690 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:06.025747 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:06.025829 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:47:06.026243 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:06.026940 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:06.028528 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:06.029267 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:06.029411 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:06.029485 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:06.029747 (+ 262us) server_negotiation.cc:299] Negotiation successful
0619 18:47:06.029870 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 18:47:06.030578 6301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:06.025030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.67:38951 (local address 127.3.243.65:45537)
0619 18:47:06.025239 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:06.025245 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:06.025843 (+ 598us) server_negotiation.cc:407] Connection header received
0619 18:47:06.026002 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:06.026006 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:06.026063 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:06.026138 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:47:06.026604 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:06.027276 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:06.028937 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:06.029707 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:06.029851 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:06.029921 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:06.030221 (+ 300us) server_negotiation.cc:299] Negotiation successful
0619 18:47:06.030343 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 18:47:06.030740 6314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:06.025027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.65:45537 (local address 127.3.243.67:38951)
0619 18:47:06.025727 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:06.025745 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:06.025856 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:06.026171 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:06.026175 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:06.026199 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:47:06.026459 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:06.026467 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:06.027409 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:06.027413 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:06.028786 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:06.028798 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:06.029902 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:06.029908 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:06.030079 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:06.030101 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 18:47:06.030180 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 18:47:06.030339 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"mutex_wait_us":3,"thread_start_us":415,"threads_started":1}
I20260619 18:47:06.031306 6097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50b6ac7278c84239bb7ec4c97ee2eb78" candidate_uuid: "efb278fc23954c9d86d8c54f68b6462e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3dbb92deec94038b08d065aa47a1bf1" is_pre_election: true
I20260619 18:47:06.031637 5958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50b6ac7278c84239bb7ec4c97ee2eb78" candidate_uuid: "efb278fc23954c9d86d8c54f68b6462e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b146718b0a3244aca8d5d949e5d8f13e" is_pre_election: true
I20260619 18:47:06.031888 6097 raft_consensus.cc:2454] T 50b6ac7278c84239bb7ec4c97ee2eb78 P e3dbb92deec94038b08d065aa47a1bf1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efb278fc23954c9d86d8c54f68b6462e in term 0.
I20260619 18:47:06.032140 5958 raft_consensus.cc:2454] T 50b6ac7278c84239bb7ec4c97ee2eb78 P b146718b0a3244aca8d5d949e5d8f13e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efb278fc23954c9d86d8c54f68b6462e in term 0.
I20260619 18:47:06.032629 6169 leader_election.cc:302] T 50b6ac7278c84239bb7ec4c97ee2eb78 P efb278fc23954c9d86d8c54f68b6462e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e3dbb92deec94038b08d065aa47a1bf1, efb278fc23954c9d86d8c54f68b6462e; no voters:
I20260619 18:47:06.032989 6313 raft_consensus.cc:2781] T 50b6ac7278c84239bb7ec4c97ee2eb78 P efb278fc23954c9d86d8c54f68b6462e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:47:06.033113 6313 raft_consensus.cc:482] T 50b6ac7278c84239bb7ec4c97ee2eb78 P efb278fc23954c9d86d8c54f68b6462e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:47:06.033205 6313 raft_consensus.cc:3037] T 50b6ac7278c84239bb7ec4c97ee2eb78 P efb278fc23954c9d86d8c54f68b6462e [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:47:06.035555 6313 raft_consensus.cc:504] T 50b6ac7278c84239bb7ec4c97ee2eb78 P efb278fc23954c9d86d8c54f68b6462e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b146718b0a3244aca8d5d949e5d8f13e" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 45537 } } peers { permanent_uuid: "e3dbb92deec94038b08d065aa47a1bf1" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 35763 } } peers { permanent_uuid: "efb278fc23954c9d86d8c54f68b6462e" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 35465 } }
I20260619 18:47:06.036170 6313 leader_election.cc:288] T 50b6ac7278c84239bb7ec4c97ee2eb78 P efb278fc23954c9d86d8c54f68b6462e [CANDIDATE]: Term 1 election: Requested vote from peers b146718b0a3244aca8d5d949e5d8f13e (127.3.243.65:45537), e3dbb92deec94038b08d065aa47a1bf1 (127.3.243.66:35763)
I20260619 18:47:06.036509 5958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50b6ac7278c84239bb7ec4c97ee2eb78" candidate_uuid: "efb278fc23954c9d86d8c54f68b6462e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b146718b0a3244aca8d5d949e5d8f13e"
I20260619 18:47:06.036636 6097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50b6ac7278c84239bb7ec4c97ee2eb78" candidate_uuid: "efb278fc23954c9d86d8c54f68b6462e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3dbb92deec94038b08d065aa47a1bf1"
I20260619 18:47:06.036724 5958 raft_consensus.cc:3037] T 50b6ac7278c84239bb7ec4c97ee2eb78 P b146718b0a3244aca8d5d949e5d8f13e [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:47:06.036824 6097 raft_consensus.cc:3037] T 50b6ac7278c84239bb7ec4c97ee2eb78 P e3dbb92deec94038b08d065aa47a1bf1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:47:06.039795 5958 raft_consensus.cc:2454] T 50b6ac7278c84239bb7ec4c97ee2eb78 P b146718b0a3244aca8d5d949e5d8f13e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efb278fc23954c9d86d8c54f68b6462e in term 1.
I20260619 18:47:06.039795 6097 raft_consensus.cc:2454] T 50b6ac7278c84239bb7ec4c97ee2eb78 P e3dbb92deec94038b08d065aa47a1bf1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efb278fc23954c9d86d8c54f68b6462e in term 1.
I20260619 18:47:06.040213 6170 leader_election.cc:302] T 50b6ac7278c84239bb7ec4c97ee2eb78 P efb278fc23954c9d86d8c54f68b6462e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b146718b0a3244aca8d5d949e5d8f13e, efb278fc23954c9d86d8c54f68b6462e; no voters:
I20260619 18:47:06.040494 6313 raft_consensus.cc:2781] T 50b6ac7278c84239bb7ec4c97ee2eb78 P efb278fc23954c9d86d8c54f68b6462e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:47:06.040877 6313 raft_consensus.cc:686] T 50b6ac7278c84239bb7ec4c97ee2eb78 P efb278fc23954c9d86d8c54f68b6462e [term 1 LEADER]: Becoming Leader. State: Replica: efb278fc23954c9d86d8c54f68b6462e, State: Running, Role: LEADER
I20260619 18:47:06.041304 6313 consensus_queue.cc:237] T 50b6ac7278c84239bb7ec4c97ee2eb78 P efb278fc23954c9d86d8c54f68b6462e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b146718b0a3244aca8d5d949e5d8f13e" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 45537 } } peers { permanent_uuid: "e3dbb92deec94038b08d065aa47a1bf1" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 35763 } } peers { permanent_uuid: "efb278fc23954c9d86d8c54f68b6462e" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 35465 } }
I20260619 18:47:06.045567 5801 catalog_manager.cc:5495] T 50b6ac7278c84239bb7ec4c97ee2eb78 P efb278fc23954c9d86d8c54f68b6462e reported cstate change: term changed from 0 to 1, leader changed from <none> to efb278fc23954c9d86d8c54f68b6462e (127.3.243.67). New cstate: current_term: 1 leader_uuid: "efb278fc23954c9d86d8c54f68b6462e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b146718b0a3244aca8d5d949e5d8f13e" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 45537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3dbb92deec94038b08d065aa47a1bf1" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 35763 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efb278fc23954c9d86d8c54f68b6462e" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 35465 } health_report { overall_health: HEALTHY } } }
Jun 19 18:47:06 dist-test-slave-m9wg krb5kdc[5756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894824, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.126@KRBTEST.COM for kudu/127.3.243.126@KRBTEST.COM
I20260619 18:47:06.152012 5895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:06.134385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48420 (local address 127.3.243.126:39867)
0619 18:47:06.134544 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:06.134548 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:06.135072 (+ 524us) server_negotiation.cc:407] Connection header received
0619 18:47:06.135237 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:06.135242 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:06.135306 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:06.135385 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:06.136564 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:06.137275 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:06.138429 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:06.138835 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:06.138936 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:06.146854 (+ 7918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:06.146859 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:06.146863 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:06.146892 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:06.149396 (+ 2504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:06.150067 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:06.150073 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:06.150076 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:06.150198 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:06.150550 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:06.150555 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:06.150558 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:06.150844 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:06.150982 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:06.151390 (+ 408us) server_negotiation.cc:299] Negotiation successful
0619 18:47:06.151700 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":116,"server-negotiator.queue_time_us":52}
I20260619 18:47:06.152017 6322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:06.134426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:39867 (local address 127.0.0.1:48420)
0619 18:47:06.134975 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:06.134990 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:06.135129 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:06.135435 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:06.135439 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:06.136124 (+ 685us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:47:06.136423 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:06.136433 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:06.137393 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:06.137397 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:06.138303 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:06.138314 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:06.139001 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:06.139007 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:06.139154 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:06.144210 (+ 5056us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:47:06.144229 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:47:06.146654 (+ 2425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:47:06.149536 (+ 2882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:06.149550 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:06.149558 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:06.149959 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:06.150313 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:06.150318 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:06.150322 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:06.150455 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:06.150973 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:47:06.150982 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:47:06.151146 (+ 164us) client_negotiation.cc:769] Sending connection context
0619 18:47:06.151285 (+ 139us) client_negotiation.cc:240] Negotiation successful
0619 18:47:06.151565 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":292,"threads_started":1}
I20260619 18:47:06.157208 5801 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.3.243.126@KRBTEST.COM'} at 127.0.0.1:48420:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 18:47:06.158195 5801 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:47:06.164598 6216 tablet_service.cc:1467] Processing CreateTablet for tablet d3885be552d143e1af0ebe4df4a84808 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d744b1ac10ae4a5fad5b4085d2018b31]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 18:47:06.164400 6077 tablet_service.cc:1467] Processing CreateTablet for tablet d3885be552d143e1af0ebe4df4a84808 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d744b1ac10ae4a5fad5b4085d2018b31]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 18:47:06.164377 5938 tablet_service.cc:1467] Processing CreateTablet for tablet d3885be552d143e1af0ebe4df4a84808 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d744b1ac10ae4a5fad5b4085d2018b31]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 18:47:06.165335 6216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3885be552d143e1af0ebe4df4a84808. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:47:06.165359 6077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3885be552d143e1af0ebe4df4a84808. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:47:06.165359 5938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3885be552d143e1af0ebe4df4a84808. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:47:06.170298 6307 tablet_bootstrap.cc:492] T d3885be552d143e1af0ebe4df4a84808 P efb278fc23954c9d86d8c54f68b6462e: Bootstrap starting.
I20260619 18:47:06.171821 6307 tablet_bootstrap.cc:654] T d3885be552d143e1af0ebe4df4a84808 P efb278fc23954c9d86d8c54f68b6462e: Neither blocks nor log segments found. Creating new log.
I20260619 18:47:06.172238 6305 tablet_bootstrap.cc:492] T d3885be552d143e1af0ebe4df4a84808 P b146718b0a3244aca8d5d949e5d8f13e: Bootstrap starting.
I20260619 18:47:06.172900 6306 tablet_bootstrap.cc:492] T d3885be552d143e1af0ebe4df4a84808 P e3dbb92deec94038b08d065aa47a1bf1: Bootstrap starting.
I20260619 18:47:06.173772 6305 tablet_bootstrap.cc:654] T d3885be552d143e1af0ebe4df4a84808 P b146718b0a3244aca8d5d949e5d8f13e: Neither blocks nor log segments found. Creating new log.
I20260619 18:47:06.174604 6306 tablet_bootstrap.cc:654] T d3885be552d143e1af0ebe4df4a84808 P e3dbb92deec94038b08d065aa47a1bf1: Neither blocks nor log segments found. Creating new log.
I20260619 18:47:06.183736 6307 tablet_bootstrap.cc:492] T d3885be552d143e1af0ebe4df4a84808 P efb278fc23954c9d86d8c54f68b6462e: No bootstrap required, opened a new log
I20260619 18:47:06.183954 6307 ts_tablet_manager.cc:1397] T d3885be552d143e1af0ebe4df4a84808 P efb278fc23954c9d86d8c54f68b6462e: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.007s
I20260619 18:47:06.184756 6307 raft_consensus.cc:348] T d3885be552d143e1af0ebe4df4a84808 P efb278fc23954c9d86d8c54f68b6462e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b146718b0a3244aca8d5d949e5d8f13e" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 45537 } } peers { permanent_uuid: "e3dbb92deec94038b08d065aa47a1bf1" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 35763 } } peers { permanent_uuid: "efb278fc23954c9d86d8c54f68b6462e" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 35465 } }
I20260619 18:47:06.185091 6307 raft_consensus.cc:374] T d3885be552d143e1af0ebe4df4a84808 P efb278fc23954c9d86d8c54f68b6462e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:47:06.185205 6307 raft_consensus.cc:729] T d3885be552d143e1af0ebe4df4a84808 P efb278fc23954c9d86d8c54f68b6462e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efb278fc23954c9d86d8c54f68b6462e, State: Initialized, Role: FOLLOWER
I20260619 18:47:06.185483 6307 consensus_queue.cc:260] T d3885be552d143e1af0ebe4df4a84808 P efb278fc23954c9d86d8c54f68b6462e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b146718b0a3244aca8d5d949e5d8f13e" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 45537 } } peers { permanent_uuid: "e3dbb92deec94038b08d065aa47a1bf1" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 35763 } } peers { permanent_uuid: "efb278fc23954c9d86d8c54f68b6462e" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 35465 } }
I20260619 18:47:06.186144 6307 ts_tablet_manager.cc:1428] T d3885be552d143e1af0ebe4df4a84808 P efb278fc23954c9d86d8c54f68b6462e: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260619 18:47:06.186045 6313 tablet_replica.cc:439] T d3885be552d143e1af0ebe4df4a84808 P efb278fc23954c9d86d8c54f68b6462e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b146718b0a3244aca8d5d949e5d8f13e" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 45537 } } peers { permanent_uuid: "e3dbb92deec94038b08d065aa47a1bf1" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 35763 } } peers { permanent_uuid: "efb278fc23954c9d86d8c54f68b6462e" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 35465 } } }
I20260619 18:47:06.186396 6313 tablet_replica.cc:442] T d3885be552d143e1af0ebe4df4a84808 P efb278fc23954c9d86d8c54f68b6462e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 18:47:06.188707 6306 tablet_bootstrap.cc:492] T d3885be552d143e1af0ebe4df4a84808 P e3dbb92deec94038b08d065aa47a1bf1: No bootstrap required, opened a new log
I20260619 18:47:06.188886 6306 ts_tablet_manager.cc:1397] T d3885be552d143e1af0ebe4df4a84808 P e3dbb92deec94038b08d065aa47a1bf1: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.009s
I20260619 18:47:06.188990 6305 tablet_bootstrap.cc:492] T d3885be552d143e1af0ebe4df4a84808 P b146718b0a3244aca8d5d949e5d8f13e: No bootstrap required, opened a new log
I20260619 18:47:06.189160 6305 ts_tablet_manager.cc:1397] T d3885be552d143e1af0ebe4df4a84808 P b146718b0a3244aca8d5d949e5d8f13e: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.012s
I20260619 18:47:06.189715 6306 raft_consensus.cc:348] T d3885be552d143e1af0ebe4df4a84808 P e3dbb92deec94038b08d065aa47a1bf1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b146718b0a3244aca8d5d949e5d8f13e" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 45537 } } peers { permanent_uuid: "e3dbb92deec94038b08d065aa47a1bf1" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 35763 } } peers { permanent_uuid: "efb278fc23954c9d86d8c54f68b6462e" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 35465 } }
I20260619 18:47:06.189986 6306 raft_consensus.cc:374] T d3885be552d143e1af0ebe4df4a84808 P e3dbb92deec94038b08d065aa47a1bf1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:47:06.189893 6305 raft_consensus.cc:348] T d3885be552d143e1af0ebe4df4a84808 P b146718b0a3244aca8d5d949e5d8f13e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b146718b0a3244aca8d5d949e5d8f13e" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 45537 } } peers { permanent_uuid: "e3dbb92deec94038b08d065aa47a1bf1" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 35763 } } peers { permanent_uuid: "efb278fc23954c9d86d8c54f68b6462e" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 35465 } }
I20260619 18:47:06.190091 6306 raft_consensus.cc:729] T d3885be552d143e1af0ebe4df4a84808 P e3dbb92deec94038b08d065aa47a1bf1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3dbb92deec94038b08d065aa47a1bf1, State: Initialized, Role: FOLLOWER
I20260619 18:47:06.190132 6305 raft_consensus.cc:374] T d3885be552d143e1af0ebe4df4a84808 P b146718b0a3244aca8d5d949e5d8f13e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:47:06.190203 6305 raft_consensus.cc:729] T d3885be552d143e1af0ebe4df4a84808 P b146718b0a3244aca8d5d949e5d8f13e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b146718b0a3244aca8d5d949e5d8f13e, State: Initialized, Role: FOLLOWER
I20260619 18:47:06.190388 6306 consensus_queue.cc:260] T d3885be552d143e1af0ebe4df4a84808 P e3dbb92deec94038b08d065aa47a1bf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b146718b0a3244aca8d5d949e5d8f13e" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 45537 } } peers { permanent_uuid: "e3dbb92deec94038b08d065aa47a1bf1" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 35763 } } peers { permanent_uuid: "efb278fc23954c9d86d8c54f68b6462e" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 35465 } }
I20260619 18:47:06.190410 6305 consensus_queue.cc:260] T d3885be552d143e1af0ebe4df4a84808 P b146718b0a3244aca8d5d949e5d8f13e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b146718b0a3244aca8d5d949e5d8f13e" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 45537 } } peers { permanent_uuid: "e3dbb92deec94038b08d065aa47a1bf1" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 35763 } } peers { permanent_uuid: "efb278fc23954c9d86d8c54f68b6462e" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 35465 } }
I20260619 18:47:06.190980 6305 ts_tablet_manager.cc:1428] T d3885be552d143e1af0ebe4df4a84808 P b146718b0a3244aca8d5d949e5d8f13e: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260619 18:47:06.190874 6311 tablet_replica.cc:439] T d3885be552d143e1af0ebe4df4a84808 P b146718b0a3244aca8d5d949e5d8f13e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b146718b0a3244aca8d5d949e5d8f13e" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 45537 } } peers { permanent_uuid: "e3dbb92deec94038b08d065aa47a1bf1" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 35763 } } peers { permanent_uuid: "efb278fc23954c9d86d8c54f68b6462e" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 35465 } } }
I20260619 18:47:06.191258 6311 tablet_replica.cc:442] T d3885be552d143e1af0ebe4df4a84808 P b146718b0a3244aca8d5d949e5d8f13e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 18:47:06.191498 6306 ts_tablet_manager.cc:1428] T d3885be552d143e1af0ebe4df4a84808 P e3dbb92deec94038b08d065aa47a1bf1: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260619 18:47:06.191656 6312 tablet_replica.cc:439] T d3885be552d143e1af0ebe4df4a84808 P e3dbb92deec94038b08d065aa47a1bf1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b146718b0a3244aca8d5d949e5d8f13e" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 45537 } } peers { permanent_uuid: "e3dbb92deec94038b08d065aa47a1bf1" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 35763 } } peers { permanent_uuid: "efb278fc23954c9d86d8c54f68b6462e" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 35465 } } }
I20260619 18:47:06.191969 6312 tablet_replica.cc:442] T d3885be552d143e1af0ebe4df4a84808 P e3dbb92deec94038b08d065aa47a1bf1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 18:47:06.395273 6313 raft_consensus.cc:482] T d3885be552d143e1af0ebe4df4a84808 P efb278fc23954c9d86d8c54f68b6462e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:47:06.395466 6313 raft_consensus.cc:504] T d3885be552d143e1af0ebe4df4a84808 P efb278fc23954c9d86d8c54f68b6462e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b146718b0a3244aca8d5d949e5d8f13e" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 45537 } } peers { permanent_uuid: "e3dbb92deec94038b08d065aa47a1bf1" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 35763 } } peers { permanent_uuid: "efb278fc23954c9d86d8c54f68b6462e" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 35465 } }
I20260619 18:47:06.396122 6313 leader_election.cc:288] T d3885be552d143e1af0ebe4df4a84808 P efb278fc23954c9d86d8c54f68b6462e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b146718b0a3244aca8d5d949e5d8f13e (127.3.243.65:45537), e3dbb92deec94038b08d065aa47a1bf1 (127.3.243.66:35763)
I20260619 18:47:06.396803 5958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3885be552d143e1af0ebe4df4a84808" candidate_uuid: "efb278fc23954c9d86d8c54f68b6462e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b146718b0a3244aca8d5d949e5d8f13e" is_pre_election: true
I20260619 18:47:06.396813 6097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3885be552d143e1af0ebe4df4a84808" candidate_uuid: "efb278fc23954c9d86d8c54f68b6462e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3dbb92deec94038b08d065aa47a1bf1" is_pre_election: true
I20260619 18:47:06.397053 5958 raft_consensus.cc:2454] T d3885be552d143e1af0ebe4df4a84808 P b146718b0a3244aca8d5d949e5d8f13e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efb278fc23954c9d86d8c54f68b6462e in term 0.
I20260619 18:47:06.397060 6097 raft_consensus.cc:2454] T d3885be552d143e1af0ebe4df4a84808 P e3dbb92deec94038b08d065aa47a1bf1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efb278fc23954c9d86d8c54f68b6462e in term 0.
I20260619 18:47:06.397459 6170 leader_election.cc:302] T d3885be552d143e1af0ebe4df4a84808 P efb278fc23954c9d86d8c54f68b6462e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b146718b0a3244aca8d5d949e5d8f13e, efb278fc23954c9d86d8c54f68b6462e; no voters:
I20260619 18:47:06.397746 6313 raft_consensus.cc:2781] T d3885be552d143e1af0ebe4df4a84808 P efb278fc23954c9d86d8c54f68b6462e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:47:06.397853 6313 raft_consensus.cc:482] T d3885be552d143e1af0ebe4df4a84808 P efb278fc23954c9d86d8c54f68b6462e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:47:06.397969 6313 raft_consensus.cc:3037] T d3885be552d143e1af0ebe4df4a84808 P efb278fc23954c9d86d8c54f68b6462e [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:47:06.399986 6313 raft_consensus.cc:504] T d3885be552d143e1af0ebe4df4a84808 P efb278fc23954c9d86d8c54f68b6462e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b146718b0a3244aca8d5d949e5d8f13e" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 45537 } } peers { permanent_uuid: "e3dbb92deec94038b08d065aa47a1bf1" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 35763 } } peers { permanent_uuid: "efb278fc23954c9d86d8c54f68b6462e" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 35465 } }
I20260619 18:47:06.400550 6313 leader_election.cc:288] T d3885be552d143e1af0ebe4df4a84808 P efb278fc23954c9d86d8c54f68b6462e [CANDIDATE]: Term 1 election: Requested vote from peers b146718b0a3244aca8d5d949e5d8f13e (127.3.243.65:45537), e3dbb92deec94038b08d065aa47a1bf1 (127.3.243.66:35763)
I20260619 18:47:06.400933 5958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3885be552d143e1af0ebe4df4a84808" candidate_uuid: "efb278fc23954c9d86d8c54f68b6462e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b146718b0a3244aca8d5d949e5d8f13e"
I20260619 18:47:06.400985 6097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3885be552d143e1af0ebe4df4a84808" candidate_uuid: "efb278fc23954c9d86d8c54f68b6462e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3dbb92deec94038b08d065aa47a1bf1"
I20260619 18:47:06.401132 5958 raft_consensus.cc:3037] T d3885be552d143e1af0ebe4df4a84808 P b146718b0a3244aca8d5d949e5d8f13e [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:47:06.401162 6097 raft_consensus.cc:3037] T d3885be552d143e1af0ebe4df4a84808 P e3dbb92deec94038b08d065aa47a1bf1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:47:06.403055 6097 raft_consensus.cc:2454] T d3885be552d143e1af0ebe4df4a84808 P e3dbb92deec94038b08d065aa47a1bf1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efb278fc23954c9d86d8c54f68b6462e in term 1.
I20260619 18:47:06.403118 5958 raft_consensus.cc:2454] T d3885be552d143e1af0ebe4df4a84808 P b146718b0a3244aca8d5d949e5d8f13e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efb278fc23954c9d86d8c54f68b6462e in term 1.
I20260619 18:47:06.403406 6169 leader_election.cc:302] T d3885be552d143e1af0ebe4df4a84808 P efb278fc23954c9d86d8c54f68b6462e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e3dbb92deec94038b08d065aa47a1bf1, efb278fc23954c9d86d8c54f68b6462e; no voters:
I20260619 18:47:06.403663 6313 raft_consensus.cc:2781] T d3885be552d143e1af0ebe4df4a84808 P efb278fc23954c9d86d8c54f68b6462e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:47:06.403823 6313 raft_consensus.cc:686] T d3885be552d143e1af0ebe4df4a84808 P efb278fc23954c9d86d8c54f68b6462e [term 1 LEADER]: Becoming Leader. State: Replica: efb278fc23954c9d86d8c54f68b6462e, State: Running, Role: LEADER
I20260619 18:47:06.404078 6313 consensus_queue.cc:237] T d3885be552d143e1af0ebe4df4a84808 P efb278fc23954c9d86d8c54f68b6462e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b146718b0a3244aca8d5d949e5d8f13e" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 45537 } } peers { permanent_uuid: "e3dbb92deec94038b08d065aa47a1bf1" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 35763 } } peers { permanent_uuid: "efb278fc23954c9d86d8c54f68b6462e" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 35465 } }
I20260619 18:47:06.405656 6315 tablet_replica.cc:439] T d3885be552d143e1af0ebe4df4a84808 P efb278fc23954c9d86d8c54f68b6462e: TxnStatusTablet state changed. Reason: New leader efb278fc23954c9d86d8c54f68b6462e. Latest consensus state: current_term: 1 leader_uuid: "efb278fc23954c9d86d8c54f68b6462e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b146718b0a3244aca8d5d949e5d8f13e" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 45537 } } peers { permanent_uuid: "e3dbb92deec94038b08d065aa47a1bf1" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 35763 } } peers { permanent_uuid: "efb278fc23954c9d86d8c54f68b6462e" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 35465 } } }
I20260619 18:47:06.405907 6315 tablet_replica.cc:442] T d3885be552d143e1af0ebe4df4a84808 P efb278fc23954c9d86d8c54f68b6462e: This TxnStatusTablet replica's current role is: LEADER
I20260619 18:47:06.406490 6327 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 18:47:06.406790 6327 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 18:47:06.407155 5799 catalog_manager.cc:5495] T d3885be552d143e1af0ebe4df4a84808 P efb278fc23954c9d86d8c54f68b6462e reported cstate change: term changed from 0 to 1, leader changed from <none> to efb278fc23954c9d86d8c54f68b6462e (127.3.243.67). New cstate: current_term: 1 leader_uuid: "efb278fc23954c9d86d8c54f68b6462e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b146718b0a3244aca8d5d949e5d8f13e" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 45537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3dbb92deec94038b08d065aa47a1bf1" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 35763 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efb278fc23954c9d86d8c54f68b6462e" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 35465 } health_report { overall_health: HEALTHY } } }
I20260619 18:47:06.427775 6322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:06.421953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.67:35465 (local address 127.0.0.1:39986)
0619 18:47:06.422165 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:06.422186 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:06.422320 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:06.422853 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:06.422858 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:06.422871 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 18:47:06.423064 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:06.423074 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:06.424125 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:06.424130 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:06.425268 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:06.425281 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:06.426064 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:06.426070 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:06.426224 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:06.426277 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 18:47:06.427236 (+ 959us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 18:47:06.427256 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 18:47:06.427340 (+ 84us) client_negotiation.cc:240] Negotiation successful
0619 18:47:06.427488 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260619 18:47:06.427868 6330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:06.422033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39986 (local address 127.3.243.67:35465)
0619 18:47:06.422518 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:06.422526 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:06.422543 (+ 17us) server_negotiation.cc:407] Connection header received
0619 18:47:06.422611 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:06.422615 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:06.422692 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:06.422813 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:47:06.423208 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:06.423998 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:06.425447 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:06.425910 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:06.426023 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:06.426490 (+ 467us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:47:06.427131 (+ 641us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:47:06.427256 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:06.427364 (+ 108us) server_negotiation.cc:299] Negotiation successful
0619 18:47:06.427454 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":215,"threads_started":1}
I20260619 18:47:06.436077 5958 raft_consensus.cc:1261] T d3885be552d143e1af0ebe4df4a84808 P b146718b0a3244aca8d5d949e5d8f13e [term 1 FOLLOWER]: Refusing update from remote peer efb278fc23954c9d86d8c54f68b6462e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:47:06.436077 6097 raft_consensus.cc:1261] T d3885be552d143e1af0ebe4df4a84808 P e3dbb92deec94038b08d065aa47a1bf1 [term 1 FOLLOWER]: Refusing update from remote peer efb278fc23954c9d86d8c54f68b6462e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:47:06.437111 6315 consensus_queue.cc:1035] T d3885be552d143e1af0ebe4df4a84808 P efb278fc23954c9d86d8c54f68b6462e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b146718b0a3244aca8d5d949e5d8f13e" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 45537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:47:06.437482 6313 consensus_queue.cc:1035] T d3885be552d143e1af0ebe4df4a84808 P efb278fc23954c9d86d8c54f68b6462e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3dbb92deec94038b08d065aa47a1bf1" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 35763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:47:06.448208 6312 tablet_replica.cc:439] T d3885be552d143e1af0ebe4df4a84808 P e3dbb92deec94038b08d065aa47a1bf1: TxnStatusTablet state changed. Reason: New leader efb278fc23954c9d86d8c54f68b6462e. Latest consensus state: current_term: 1 leader_uuid: "efb278fc23954c9d86d8c54f68b6462e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b146718b0a3244aca8d5d949e5d8f13e" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 45537 } } peers { permanent_uuid: "e3dbb92deec94038b08d065aa47a1bf1" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 35763 } } peers { permanent_uuid: "efb278fc23954c9d86d8c54f68b6462e" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 35465 } } }
I20260619 18:47:06.448513 6312 tablet_replica.cc:442] T d3885be552d143e1af0ebe4df4a84808 P e3dbb92deec94038b08d065aa47a1bf1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 18:47:06.450749 6331 mvcc.cc:205] Tried to move back new op lower bound from 7298641209072390144 to 7298641208953335808. Current Snapshot: MvccSnapshot[applied={T|T < 7298641209072390144}]
I20260619 18:47:06.450723 6311 tablet_replica.cc:439] T d3885be552d143e1af0ebe4df4a84808 P b146718b0a3244aca8d5d949e5d8f13e: TxnStatusTablet state changed. Reason: New leader efb278fc23954c9d86d8c54f68b6462e. Latest consensus state: current_term: 1 leader_uuid: "efb278fc23954c9d86d8c54f68b6462e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b146718b0a3244aca8d5d949e5d8f13e" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 45537 } } peers { permanent_uuid: "e3dbb92deec94038b08d065aa47a1bf1" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 35763 } } peers { permanent_uuid: "efb278fc23954c9d86d8c54f68b6462e" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 35465 } } }
I20260619 18:47:06.450984 6311 tablet_replica.cc:442] T d3885be552d143e1af0ebe4df4a84808 P b146718b0a3244aca8d5d949e5d8f13e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 18:47:06.452227 6333 mvcc.cc:205] Tried to move back new op lower bound from 7298641209072390144 to 7298641208953335808. Current Snapshot: MvccSnapshot[applied={T|T < 7298641209072390144}]
I20260619 18:47:06.453495 6315 tablet_replica.cc:439] T d3885be552d143e1af0ebe4df4a84808 P efb278fc23954c9d86d8c54f68b6462e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "efb278fc23954c9d86d8c54f68b6462e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b146718b0a3244aca8d5d949e5d8f13e" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 45537 } } peers { permanent_uuid: "e3dbb92deec94038b08d065aa47a1bf1" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 35763 } } peers { permanent_uuid: "efb278fc23954c9d86d8c54f68b6462e" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 35465 } } }
I20260619 18:47:06.453765 6315 tablet_replica.cc:442] T d3885be552d143e1af0ebe4df4a84808 P efb278fc23954c9d86d8c54f68b6462e: This TxnStatusTablet replica's current role is: LEADER
I20260619 18:47:06.453970 6315 tablet_replica.cc:439] T d3885be552d143e1af0ebe4df4a84808 P efb278fc23954c9d86d8c54f68b6462e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "efb278fc23954c9d86d8c54f68b6462e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b146718b0a3244aca8d5d949e5d8f13e" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 45537 } } peers { permanent_uuid: "e3dbb92deec94038b08d065aa47a1bf1" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 35763 } } peers { permanent_uuid: "efb278fc23954c9d86d8c54f68b6462e" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 35465 } } }
I20260619 18:47:06.454180 6315 tablet_replica.cc:442] T d3885be552d143e1af0ebe4df4a84808 P efb278fc23954c9d86d8c54f68b6462e: This TxnStatusTablet replica's current role is: LEADER
I20260619 18:47:06.465612 6312 tablet_replica.cc:439] T d3885be552d143e1af0ebe4df4a84808 P e3dbb92deec94038b08d065aa47a1bf1: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "efb278fc23954c9d86d8c54f68b6462e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b146718b0a3244aca8d5d949e5d8f13e" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 45537 } } peers { permanent_uuid: "e3dbb92deec94038b08d065aa47a1bf1" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 35763 } } peers { permanent_uuid: "efb278fc23954c9d86d8c54f68b6462e" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 35465 } } }
I20260619 18:47:06.465823 6312 tablet_replica.cc:442] T d3885be552d143e1af0ebe4df4a84808 P e3dbb92deec94038b08d065aa47a1bf1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 18:47:06.471865 6311 tablet_replica.cc:439] T d3885be552d143e1af0ebe4df4a84808 P b146718b0a3244aca8d5d949e5d8f13e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "efb278fc23954c9d86d8c54f68b6462e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b146718b0a3244aca8d5d949e5d8f13e" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 45537 } } peers { permanent_uuid: "e3dbb92deec94038b08d065aa47a1bf1" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 35763 } } peers { permanent_uuid: "efb278fc23954c9d86d8c54f68b6462e" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 35465 } } }
I20260619 18:47:06.472131 6311 tablet_replica.cc:442] T d3885be552d143e1af0ebe4df4a84808 P b146718b0a3244aca8d5d949e5d8f13e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 18:47:06.481629 6330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:06.475808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39994 (local address 127.3.243.67:35465)
0619 18:47:06.475985 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:06.475991 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:06.476747 (+ 756us) server_negotiation.cc:407] Connection header received
0619 18:47:06.477037 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:06.477041 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:06.477113 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:06.477210 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:47:06.477651 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:06.478506 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:06.479953 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:06.480453 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:06.480568 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:06.480925 (+ 357us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:47:06.481038 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:47:06.481112 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:06.481247 (+ 135us) server_negotiation.cc:299] Negotiation successful
0619 18:47:06.481329 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 18:47:06.498198 6330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:06.492465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.67:36525 (local address 127.3.243.67:35465)
0619 18:47:06.492608 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:06.492612 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:06.493213 (+ 601us) server_negotiation.cc:407] Connection header received
0619 18:47:06.493397 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:06.493402 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:06.493473 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:06.493550 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:47:06.494116 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:06.494993 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:06.496541 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:06.496958 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:06.497067 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:06.497481 (+ 414us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:47:06.497578 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:47:06.497647 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:06.497764 (+ 117us) server_negotiation.cc:299] Negotiation successful
0619 18:47:06.497834 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":38}
I20260619 18:47:06.498284 6343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:06.492521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.67:35465 (local address 127.3.243.67:36525)
0619 18:47:06.493121 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:06.493135 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:06.493272 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:06.493572 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:06.493576 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:06.493586 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 18:47:06.493940 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:06.493973 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:06.495110 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:06.495115 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:06.496420 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:06.496433 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:06.497124 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:06.497130 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:06.497245 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:06.497296 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 18:47:06.497661 (+ 365us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 18:47:06.497671 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 18:47:06.497739 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 18:47:06.497870 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"mutex_wait_us":18,"thread_start_us":322,"threads_started":1}
I20260619 18:47:06.507414 5958 raft_consensus.cc:1261] T 50b6ac7278c84239bb7ec4c97ee2eb78 P b146718b0a3244aca8d5d949e5d8f13e [term 1 FOLLOWER]: Refusing update from remote peer efb278fc23954c9d86d8c54f68b6462e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:47:06.507506 6096 raft_consensus.cc:1261] T 50b6ac7278c84239bb7ec4c97ee2eb78 P e3dbb92deec94038b08d065aa47a1bf1 [term 1 FOLLOWER]: Refusing update from remote peer efb278fc23954c9d86d8c54f68b6462e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:47:06.507983 6344 consensus_queue.cc:1035] T 50b6ac7278c84239bb7ec4c97ee2eb78 P efb278fc23954c9d86d8c54f68b6462e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b146718b0a3244aca8d5d949e5d8f13e" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 45537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:47:06.508342 6313 consensus_queue.cc:1035] T 50b6ac7278c84239bb7ec4c97ee2eb78 P efb278fc23954c9d86d8c54f68b6462e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3dbb92deec94038b08d065aa47a1bf1" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 35763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:47:06.518540 6332 mvcc.cc:205] Tried to move back new op lower bound from 7298641209368674304 to 7298641207468580864. Current Snapshot: MvccSnapshot[applied={T|T < 7298641209368674304}]
I20260619 18:47:06.565259 5799 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48404:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:47:06.565584 5799 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48404:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:47:06.569011 5799 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7647451422aa4ed39c38da3831664f1b: Sending DeleteTablet for 3 replicas of tablet 50b6ac7278c84239bb7ec4c97ee2eb78
I20260619 18:47:06.570020 5938 tablet_service.cc:1514] Processing DeleteTablet for tablet 50b6ac7278c84239bb7ec4c97ee2eb78 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:47:06 UTC) from {username='kudu', principal='kudu/127.3.243.126@KRBTEST.COM'} at 127.0.0.1:50528
I20260619 18:47:06.570245 6077 tablet_service.cc:1514] Processing DeleteTablet for tablet 50b6ac7278c84239bb7ec4c97ee2eb78 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:47:06 UTC) from {username='kudu', principal='kudu/127.3.243.126@KRBTEST.COM'} at 127.0.0.1:32794
I20260619 18:47:06.571262 6216 tablet_service.cc:1514] Processing DeleteTablet for tablet 50b6ac7278c84239bb7ec4c97ee2eb78 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:47:06 UTC) from {username='kudu', principal='kudu/127.3.243.126@KRBTEST.COM'} at 127.0.0.1:39980
I20260619 18:47:06.571969 5799 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48404:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:47:06.572665 5799 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:47:06.581169 6076 tablet_service.cc:1467] Processing CreateTablet for tablet 61d88a7163ab4e9792269ad926891ba4 (DEFAULT_TABLE table=test-table [id=e3fe398264b54b3c98cb203ca4c69e8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:47:06.581277 5937 tablet_service.cc:1467] Processing CreateTablet for tablet 61d88a7163ab4e9792269ad926891ba4 (DEFAULT_TABLE table=test-table [id=e3fe398264b54b3c98cb203ca4c69e8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:47:06.581679 6076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61d88a7163ab4e9792269ad926891ba4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:47:06.581831 5937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61d88a7163ab4e9792269ad926891ba4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:47:06.585001 6215 tablet_service.cc:1467] Processing CreateTablet for tablet 61d88a7163ab4e9792269ad926891ba4 (DEFAULT_TABLE table=test-table [id=e3fe398264b54b3c98cb203ca4c69e8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:47:06.585573 6215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61d88a7163ab4e9792269ad926891ba4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:47:06.588615 6352 tablet_replica.cc:330] T 50b6ac7278c84239bb7ec4c97ee2eb78 P efb278fc23954c9d86d8c54f68b6462e: stopping tablet replica
I20260619 18:47:06.588766 6305 tablet_bootstrap.cc:492] T 61d88a7163ab4e9792269ad926891ba4 P b146718b0a3244aca8d5d949e5d8f13e: Bootstrap starting.
I20260619 18:47:06.589074 6352 raft_consensus.cc:2229] T 50b6ac7278c84239bb7ec4c97ee2eb78 P efb278fc23954c9d86d8c54f68b6462e [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:47:06.590538 6305 tablet_bootstrap.cc:654] T 61d88a7163ab4e9792269ad926891ba4 P b146718b0a3244aca8d5d949e5d8f13e: Neither blocks nor log segments found. Creating new log.
I20260619 18:47:06.590894 6351 tablet_replica.cc:330] T 50b6ac7278c84239bb7ec4c97ee2eb78 P e3dbb92deec94038b08d065aa47a1bf1: stopping tablet replica
I20260619 18:47:06.591300 6351 raft_consensus.cc:2229] T 50b6ac7278c84239bb7ec4c97ee2eb78 P e3dbb92deec94038b08d065aa47a1bf1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:47:06.591667 6351 raft_consensus.cc:2258] T 50b6ac7278c84239bb7ec4c97ee2eb78 P e3dbb92deec94038b08d065aa47a1bf1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:47:06.592090 6307 tablet_bootstrap.cc:492] T 61d88a7163ab4e9792269ad926891ba4 P efb278fc23954c9d86d8c54f68b6462e: Bootstrap starting.
I20260619 18:47:06.592449 6352 raft_consensus.cc:2258] T 50b6ac7278c84239bb7ec4c97ee2eb78 P efb278fc23954c9d86d8c54f68b6462e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:47:06.593748 6307 tablet_bootstrap.cc:654] T 61d88a7163ab4e9792269ad926891ba4 P efb278fc23954c9d86d8c54f68b6462e: Neither blocks nor log segments found. Creating new log.
I20260619 18:47:06.596526 6351 ts_tablet_manager.cc:1905] T 50b6ac7278c84239bb7ec4c97ee2eb78 P e3dbb92deec94038b08d065aa47a1bf1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:47:06.596932 6306 tablet_bootstrap.cc:492] T 61d88a7163ab4e9792269ad926891ba4 P e3dbb92deec94038b08d065aa47a1bf1: Bootstrap starting.
I20260619 18:47:06.597980 6350 tablet_replica.cc:330] T 50b6ac7278c84239bb7ec4c97ee2eb78 P b146718b0a3244aca8d5d949e5d8f13e: stopping tablet replica
I20260619 18:47:06.598446 6306 tablet_bootstrap.cc:654] T 61d88a7163ab4e9792269ad926891ba4 P e3dbb92deec94038b08d065aa47a1bf1: Neither blocks nor log segments found. Creating new log.
I20260619 18:47:06.598450 6350 raft_consensus.cc:2229] T 50b6ac7278c84239bb7ec4c97ee2eb78 P b146718b0a3244aca8d5d949e5d8f13e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:47:06.598546 6352 ts_tablet_manager.cc:1905] T 50b6ac7278c84239bb7ec4c97ee2eb78 P efb278fc23954c9d86d8c54f68b6462e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:47:06.598929 6350 raft_consensus.cc:2258] T 50b6ac7278c84239bb7ec4c97ee2eb78 P b146718b0a3244aca8d5d949e5d8f13e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:47:06.600697 6305 tablet_bootstrap.cc:492] T 61d88a7163ab4e9792269ad926891ba4 P b146718b0a3244aca8d5d949e5d8f13e: No bootstrap required, opened a new log
I20260619 18:47:06.600867 6305 ts_tablet_manager.cc:1397] T 61d88a7163ab4e9792269ad926891ba4 P b146718b0a3244aca8d5d949e5d8f13e: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.002s
I20260619 18:47:06.600871 6350 ts_tablet_manager.cc:1905] T 50b6ac7278c84239bb7ec4c97ee2eb78 P b146718b0a3244aca8d5d949e5d8f13e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:47:06.601608 6305 raft_consensus.cc:348] T 61d88a7163ab4e9792269ad926891ba4 P b146718b0a3244aca8d5d949e5d8f13e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b146718b0a3244aca8d5d949e5d8f13e" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 45537 } } peers { permanent_uuid: "e3dbb92deec94038b08d065aa47a1bf1" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 35763 } } peers { permanent_uuid: "efb278fc23954c9d86d8c54f68b6462e" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 35465 } }
I20260619 18:47:06.601879 6305 raft_consensus.cc:374] T 61d88a7163ab4e9792269ad926891ba4 P b146718b0a3244aca8d5d949e5d8f13e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:47:06.601958 6305 raft_consensus.cc:729] T 61d88a7163ab4e9792269ad926891ba4 P b146718b0a3244aca8d5d949e5d8f13e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b146718b0a3244aca8d5d949e5d8f13e, State: Initialized, Role: FOLLOWER
I20260619 18:47:06.602197 6305 consensus_queue.cc:260] T 61d88a7163ab4e9792269ad926891ba4 P b146718b0a3244aca8d5d949e5d8f13e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b146718b0a3244aca8d5d949e5d8f13e" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 45537 } } peers { permanent_uuid: "e3dbb92deec94038b08d065aa47a1bf1" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 35763 } } peers { permanent_uuid: "efb278fc23954c9d86d8c54f68b6462e" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 35465 } }
I20260619 18:47:06.605054 6305 ts_tablet_manager.cc:1428] T 61d88a7163ab4e9792269ad926891ba4 P b146718b0a3244aca8d5d949e5d8f13e: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260619 18:47:06.605387 6306 tablet_bootstrap.cc:492] T 61d88a7163ab4e9792269ad926891ba4 P e3dbb92deec94038b08d065aa47a1bf1: No bootstrap required, opened a new log
I20260619 18:47:06.605528 6306 ts_tablet_manager.cc:1397] T 61d88a7163ab4e9792269ad926891ba4 P e3dbb92deec94038b08d065aa47a1bf1: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.002s
I20260619 18:47:06.606222 6306 raft_consensus.cc:348] T 61d88a7163ab4e9792269ad926891ba4 P e3dbb92deec94038b08d065aa47a1bf1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b146718b0a3244aca8d5d949e5d8f13e" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 45537 } } peers { permanent_uuid: "e3dbb92deec94038b08d065aa47a1bf1" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 35763 } } peers { permanent_uuid: "efb278fc23954c9d86d8c54f68b6462e" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 35465 } }
I20260619 18:47:06.606444 6306 raft_consensus.cc:374] T 61d88a7163ab4e9792269ad926891ba4 P e3dbb92deec94038b08d065aa47a1bf1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:47:06.606513 6306 raft_consensus.cc:729] T 61d88a7163ab4e9792269ad926891ba4 P e3dbb92deec94038b08d065aa47a1bf1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3dbb92deec94038b08d065aa47a1bf1, State: Initialized, Role: FOLLOWER
I20260619 18:47:06.606746 6306 consensus_queue.cc:260] T 61d88a7163ab4e9792269ad926891ba4 P e3dbb92deec94038b08d065aa47a1bf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b146718b0a3244aca8d5d949e5d8f13e" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 45537 } } peers { permanent_uuid: "e3dbb92deec94038b08d065aa47a1bf1" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 35763 } } peers { permanent_uuid: "efb278fc23954c9d86d8c54f68b6462e" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 35465 } }
I20260619 18:47:06.609279 6306 ts_tablet_manager.cc:1428] T 61d88a7163ab4e9792269ad926891ba4 P e3dbb92deec94038b08d065aa47a1bf1: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260619 18:47:06.611539 6307 tablet_bootstrap.cc:492] T 61d88a7163ab4e9792269ad926891ba4 P efb278fc23954c9d86d8c54f68b6462e: No bootstrap required, opened a new log
I20260619 18:47:06.611714 6307 ts_tablet_manager.cc:1397] T 61d88a7163ab4e9792269ad926891ba4 P efb278fc23954c9d86d8c54f68b6462e: Time spent bootstrapping tablet: real 0.020s user 0.000s sys 0.006s
I20260619 18:47:06.612519 6307 raft_consensus.cc:348] T 61d88a7163ab4e9792269ad926891ba4 P efb278fc23954c9d86d8c54f68b6462e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b146718b0a3244aca8d5d949e5d8f13e" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 45537 } } peers { permanent_uuid: "e3dbb92deec94038b08d065aa47a1bf1" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 35763 } } peers { permanent_uuid: "efb278fc23954c9d86d8c54f68b6462e" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 35465 } }
I20260619 18:47:06.612764 6307 raft_consensus.cc:374] T 61d88a7163ab4e9792269ad926891ba4 P efb278fc23954c9d86d8c54f68b6462e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:47:06.612843 6307 raft_consensus.cc:729] T 61d88a7163ab4e9792269ad926891ba4 P efb278fc23954c9d86d8c54f68b6462e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efb278fc23954c9d86d8c54f68b6462e, State: Initialized, Role: FOLLOWER
I20260619 18:47:06.613068 6307 consensus_queue.cc:260] T 61d88a7163ab4e9792269ad926891ba4 P efb278fc23954c9d86d8c54f68b6462e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b146718b0a3244aca8d5d949e5d8f13e" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 45537 } } peers { permanent_uuid: "e3dbb92deec94038b08d065aa47a1bf1" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 35763 } } peers { permanent_uuid: "efb278fc23954c9d86d8c54f68b6462e" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 35465 } }
I20260619 18:47:06.613442 6350 ts_tablet_manager.cc:1918] T 50b6ac7278c84239bb7ec4c97ee2eb78 P b146718b0a3244aca8d5d949e5d8f13e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 18:47:06.613598 6350 log.cc:1196] T 50b6ac7278c84239bb7ec4c97ee2eb78 P b146718b0a3244aca8d5d949e5d8f13e: Deleting WAL directory at /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/minicluster-data/ts-0/wal/wals/50b6ac7278c84239bb7ec4c97ee2eb78
I20260619 18:47:06.613715 6307 ts_tablet_manager.cc:1428] T 61d88a7163ab4e9792269ad926891ba4 P efb278fc23954c9d86d8c54f68b6462e: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260619 18:47:06.614100 6350 ts_tablet_manager.cc:1939] T 50b6ac7278c84239bb7ec4c97ee2eb78 P b146718b0a3244aca8d5d949e5d8f13e: Deleting consensus metadata
I20260619 18:47:06.615809 5788 catalog_manager.cc:4841] TS b146718b0a3244aca8d5d949e5d8f13e (127.3.243.65:45537): tablet 50b6ac7278c84239bb7ec4c97ee2eb78 (table test-table [id=d80636f301c64fc4953890c02155a21e]) successfully deleted
I20260619 18:47:06.617920 6351 ts_tablet_manager.cc:1918] T 50b6ac7278c84239bb7ec4c97ee2eb78 P e3dbb92deec94038b08d065aa47a1bf1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 18:47:06.618083 6351 log.cc:1196] T 50b6ac7278c84239bb7ec4c97ee2eb78 P e3dbb92deec94038b08d065aa47a1bf1: Deleting WAL directory at /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/minicluster-data/ts-1/wal/wals/50b6ac7278c84239bb7ec4c97ee2eb78
I20260619 18:47:06.618615 6351 ts_tablet_manager.cc:1939] T 50b6ac7278c84239bb7ec4c97ee2eb78 P e3dbb92deec94038b08d065aa47a1bf1: Deleting consensus metadata
I20260619 18:47:06.619366 6352 ts_tablet_manager.cc:1918] T 50b6ac7278c84239bb7ec4c97ee2eb78 P efb278fc23954c9d86d8c54f68b6462e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 18:47:06.619527 6352 log.cc:1196] T 50b6ac7278c84239bb7ec4c97ee2eb78 P efb278fc23954c9d86d8c54f68b6462e: Deleting WAL directory at /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/minicluster-data/ts-2/wal/wals/50b6ac7278c84239bb7ec4c97ee2eb78
I20260619 18:47:06.620056 6352 ts_tablet_manager.cc:1939] T 50b6ac7278c84239bb7ec4c97ee2eb78 P efb278fc23954c9d86d8c54f68b6462e: Deleting consensus metadata
I20260619 18:47:06.620404 5787 catalog_manager.cc:4841] TS e3dbb92deec94038b08d065aa47a1bf1 (127.3.243.66:35763): tablet 50b6ac7278c84239bb7ec4c97ee2eb78 (table test-table [id=d80636f301c64fc4953890c02155a21e]) successfully deleted
I20260619 18:47:06.621510 5787 catalog_manager.cc:4841] TS efb278fc23954c9d86d8c54f68b6462e (127.3.243.67:35465): tablet 50b6ac7278c84239bb7ec4c97ee2eb78 (table test-table [id=d80636f301c64fc4953890c02155a21e]) successfully deleted
I20260619 18:47:06.991916 6349 raft_consensus.cc:482] T 61d88a7163ab4e9792269ad926891ba4 P efb278fc23954c9d86d8c54f68b6462e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:47:06.992214 6349 raft_consensus.cc:504] T 61d88a7163ab4e9792269ad926891ba4 P efb278fc23954c9d86d8c54f68b6462e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b146718b0a3244aca8d5d949e5d8f13e" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 45537 } } peers { permanent_uuid: "e3dbb92deec94038b08d065aa47a1bf1" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 35763 } } peers { permanent_uuid: "efb278fc23954c9d86d8c54f68b6462e" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 35465 } }
I20260619 18:47:06.992990 6349 leader_election.cc:288] T 61d88a7163ab4e9792269ad926891ba4 P efb278fc23954c9d86d8c54f68b6462e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b146718b0a3244aca8d5d949e5d8f13e (127.3.243.65:45537), e3dbb92deec94038b08d065aa47a1bf1 (127.3.243.66:35763)
I20260619 18:47:06.993618 5958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61d88a7163ab4e9792269ad926891ba4" candidate_uuid: "efb278fc23954c9d86d8c54f68b6462e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b146718b0a3244aca8d5d949e5d8f13e" is_pre_election: true
I20260619 18:47:06.993607 6097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61d88a7163ab4e9792269ad926891ba4" candidate_uuid: "efb278fc23954c9d86d8c54f68b6462e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3dbb92deec94038b08d065aa47a1bf1" is_pre_election: true
I20260619 18:47:06.993867 5958 raft_consensus.cc:2454] T 61d88a7163ab4e9792269ad926891ba4 P b146718b0a3244aca8d5d949e5d8f13e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efb278fc23954c9d86d8c54f68b6462e in term 0.
I20260619 18:47:06.993886 6097 raft_consensus.cc:2454] T 61d88a7163ab4e9792269ad926891ba4 P e3dbb92deec94038b08d065aa47a1bf1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efb278fc23954c9d86d8c54f68b6462e in term 0.
I20260619 18:47:06.994333 6169 leader_election.cc:302] T 61d88a7163ab4e9792269ad926891ba4 P efb278fc23954c9d86d8c54f68b6462e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e3dbb92deec94038b08d065aa47a1bf1, efb278fc23954c9d86d8c54f68b6462e; no voters:
I20260619 18:47:06.994627 6349 raft_consensus.cc:2781] T 61d88a7163ab4e9792269ad926891ba4 P efb278fc23954c9d86d8c54f68b6462e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:47:06.994771 6349 raft_consensus.cc:482] T 61d88a7163ab4e9792269ad926891ba4 P efb278fc23954c9d86d8c54f68b6462e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:47:06.994859 6349 raft_consensus.cc:3037] T 61d88a7163ab4e9792269ad926891ba4 P efb278fc23954c9d86d8c54f68b6462e [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:47:06.997270 6349 raft_consensus.cc:504] T 61d88a7163ab4e9792269ad926891ba4 P efb278fc23954c9d86d8c54f68b6462e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b146718b0a3244aca8d5d949e5d8f13e" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 45537 } } peers { permanent_uuid: "e3dbb92deec94038b08d065aa47a1bf1" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 35763 } } peers { permanent_uuid: "efb278fc23954c9d86d8c54f68b6462e" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 35465 } }
I20260619 18:47:06.997812 6349 leader_election.cc:288] T 61d88a7163ab4e9792269ad926891ba4 P efb278fc23954c9d86d8c54f68b6462e [CANDIDATE]: Term 1 election: Requested vote from peers b146718b0a3244aca8d5d949e5d8f13e (127.3.243.65:45537), e3dbb92deec94038b08d065aa47a1bf1 (127.3.243.66:35763)
I20260619 18:47:06.998335 6097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61d88a7163ab4e9792269ad926891ba4" candidate_uuid: "efb278fc23954c9d86d8c54f68b6462e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3dbb92deec94038b08d065aa47a1bf1"
I20260619 18:47:06.998355 5958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61d88a7163ab4e9792269ad926891ba4" candidate_uuid: "efb278fc23954c9d86d8c54f68b6462e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b146718b0a3244aca8d5d949e5d8f13e"
I20260619 18:47:06.998581 6097 raft_consensus.cc:3037] T 61d88a7163ab4e9792269ad926891ba4 P e3dbb92deec94038b08d065aa47a1bf1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:47:06.998574 5958 raft_consensus.cc:3037] T 61d88a7163ab4e9792269ad926891ba4 P b146718b0a3244aca8d5d949e5d8f13e [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:47:07.001456 6097 raft_consensus.cc:2454] T 61d88a7163ab4e9792269ad926891ba4 P e3dbb92deec94038b08d065aa47a1bf1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efb278fc23954c9d86d8c54f68b6462e in term 1.
I20260619 18:47:07.001479 5958 raft_consensus.cc:2454] T 61d88a7163ab4e9792269ad926891ba4 P b146718b0a3244aca8d5d949e5d8f13e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efb278fc23954c9d86d8c54f68b6462e in term 1.
I20260619 18:47:07.002080 6170 leader_election.cc:302] T 61d88a7163ab4e9792269ad926891ba4 P efb278fc23954c9d86d8c54f68b6462e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: b146718b0a3244aca8d5d949e5d8f13e, e3dbb92deec94038b08d065aa47a1bf1, efb278fc23954c9d86d8c54f68b6462e; no voters:
I20260619 18:47:07.002403 6349 raft_consensus.cc:2781] T 61d88a7163ab4e9792269ad926891ba4 P efb278fc23954c9d86d8c54f68b6462e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:47:07.002553 6349 raft_consensus.cc:686] T 61d88a7163ab4e9792269ad926891ba4 P efb278fc23954c9d86d8c54f68b6462e [term 1 LEADER]: Becoming Leader. State: Replica: efb278fc23954c9d86d8c54f68b6462e, State: Running, Role: LEADER
I20260619 18:47:07.002836 6349 consensus_queue.cc:237] T 61d88a7163ab4e9792269ad926891ba4 P efb278fc23954c9d86d8c54f68b6462e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b146718b0a3244aca8d5d949e5d8f13e" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 45537 } } peers { permanent_uuid: "e3dbb92deec94038b08d065aa47a1bf1" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 35763 } } peers { permanent_uuid: "efb278fc23954c9d86d8c54f68b6462e" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 35465 } }
I20260619 18:47:07.005924 5799 catalog_manager.cc:5495] T 61d88a7163ab4e9792269ad926891ba4 P efb278fc23954c9d86d8c54f68b6462e reported cstate change: term changed from 0 to 1, leader changed from <none> to efb278fc23954c9d86d8c54f68b6462e (127.3.243.67). New cstate: current_term: 1 leader_uuid: "efb278fc23954c9d86d8c54f68b6462e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b146718b0a3244aca8d5d949e5d8f13e" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 45537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3dbb92deec94038b08d065aa47a1bf1" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 35763 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efb278fc23954c9d86d8c54f68b6462e" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 35465 } health_report { overall_health: HEALTHY } } }
I20260619 18:47:07.059067 5957 raft_consensus.cc:1261] T 61d88a7163ab4e9792269ad926891ba4 P b146718b0a3244aca8d5d949e5d8f13e [term 1 FOLLOWER]: Refusing update from remote peer efb278fc23954c9d86d8c54f68b6462e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:47:07.059633 6097 raft_consensus.cc:1261] T 61d88a7163ab4e9792269ad926891ba4 P e3dbb92deec94038b08d065aa47a1bf1 [term 1 FOLLOWER]: Refusing update from remote peer efb278fc23954c9d86d8c54f68b6462e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:47:07.059590 6349 consensus_queue.cc:1035] T 61d88a7163ab4e9792269ad926891ba4 P efb278fc23954c9d86d8c54f68b6462e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b146718b0a3244aca8d5d949e5d8f13e" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 45537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:47:07.060124 6349 consensus_queue.cc:1035] T 61d88a7163ab4e9792269ad926891ba4 P efb278fc23954c9d86d8c54f68b6462e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3dbb92deec94038b08d065aa47a1bf1" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 35763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 18:47:07.082487 6331 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 18:47:08.662307 5799 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48404:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:47:08.662580 5799 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48404:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:47:08.666714 5799 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7647451422aa4ed39c38da3831664f1b: Sending DeleteTablet for 3 replicas of tablet 61d88a7163ab4e9792269ad926891ba4
I20260619 18:47:08.667470 5938 tablet_service.cc:1514] Processing DeleteTablet for tablet 61d88a7163ab4e9792269ad926891ba4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:47:08 UTC) from {username='kudu', principal='kudu/127.3.243.126@KRBTEST.COM'} at 127.0.0.1:50528
I20260619 18:47:08.667642 6077 tablet_service.cc:1514] Processing DeleteTablet for tablet 61d88a7163ab4e9792269ad926891ba4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:47:08 UTC) from {username='kudu', principal='kudu/127.3.243.126@KRBTEST.COM'} at 127.0.0.1:32794
I20260619 18:47:08.667855 6215 tablet_service.cc:1514] Processing DeleteTablet for tablet 61d88a7163ab4e9792269ad926891ba4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:47:08 UTC) from {username='kudu', principal='kudu/127.3.243.126@KRBTEST.COM'} at 127.0.0.1:39980
I20260619 18:47:08.675886 6381 tablet_replica.cc:330] T 61d88a7163ab4e9792269ad926891ba4 P b146718b0a3244aca8d5d949e5d8f13e: stopping tablet replica
I20260619 18:47:08.676393 6382 tablet_replica.cc:330] T 61d88a7163ab4e9792269ad926891ba4 P e3dbb92deec94038b08d065aa47a1bf1: stopping tablet replica
I20260619 18:47:08.676659 6383 tablet_replica.cc:330] T 61d88a7163ab4e9792269ad926891ba4 P efb278fc23954c9d86d8c54f68b6462e: stopping tablet replica
I20260619 18:47:08.676637 6382 raft_consensus.cc:2229] T 61d88a7163ab4e9792269ad926891ba4 P e3dbb92deec94038b08d065aa47a1bf1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:47:08.676836 6383 raft_consensus.cc:2229] T 61d88a7163ab4e9792269ad926891ba4 P efb278fc23954c9d86d8c54f68b6462e [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:47:08.676887 6382 raft_consensus.cc:2258] T 61d88a7163ab4e9792269ad926891ba4 P e3dbb92deec94038b08d065aa47a1bf1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:47:08.677213 6383 raft_consensus.cc:2258] T 61d88a7163ab4e9792269ad926891ba4 P efb278fc23954c9d86d8c54f68b6462e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:47:08.677184 6381 raft_consensus.cc:2229] T 61d88a7163ab4e9792269ad926891ba4 P b146718b0a3244aca8d5d949e5d8f13e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:47:08.678325 6381 raft_consensus.cc:2258] T 61d88a7163ab4e9792269ad926891ba4 P b146718b0a3244aca8d5d949e5d8f13e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:47:08.678773 6383 ts_tablet_manager.cc:1905] T 61d88a7163ab4e9792269ad926891ba4 P efb278fc23954c9d86d8c54f68b6462e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:47:08.680401 6381 ts_tablet_manager.cc:1905] T 61d88a7163ab4e9792269ad926891ba4 P b146718b0a3244aca8d5d949e5d8f13e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:47:08.681032 6382 ts_tablet_manager.cc:1905] T 61d88a7163ab4e9792269ad926891ba4 P e3dbb92deec94038b08d065aa47a1bf1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:47:08.685242 6383 ts_tablet_manager.cc:1918] T 61d88a7163ab4e9792269ad926891ba4 P efb278fc23954c9d86d8c54f68b6462e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 18:47:08.685406 6383 log.cc:1196] T 61d88a7163ab4e9792269ad926891ba4 P efb278fc23954c9d86d8c54f68b6462e: Deleting WAL directory at /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/minicluster-data/ts-2/wal/wals/61d88a7163ab4e9792269ad926891ba4
I20260619 18:47:08.685843 6383 ts_tablet_manager.cc:1939] T 61d88a7163ab4e9792269ad926891ba4 P efb278fc23954c9d86d8c54f68b6462e: Deleting consensus metadata
I20260619 18:47:08.686955 5787 catalog_manager.cc:4841] TS efb278fc23954c9d86d8c54f68b6462e (127.3.243.67:35465): tablet 61d88a7163ab4e9792269ad926891ba4 (table test-table [id=e3fe398264b54b3c98cb203ca4c69e8c]) successfully deleted
I20260619 18:47:08.687232 6382 ts_tablet_manager.cc:1918] T 61d88a7163ab4e9792269ad926891ba4 P e3dbb92deec94038b08d065aa47a1bf1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 18:47:08.687350 6382 log.cc:1196] T 61d88a7163ab4e9792269ad926891ba4 P e3dbb92deec94038b08d065aa47a1bf1: Deleting WAL directory at /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/minicluster-data/ts-1/wal/wals/61d88a7163ab4e9792269ad926891ba4
I20260619 18:47:08.687701 6382 ts_tablet_manager.cc:1939] T 61d88a7163ab4e9792269ad926891ba4 P e3dbb92deec94038b08d065aa47a1bf1: Deleting consensus metadata
I20260619 18:47:08.688627 5787 catalog_manager.cc:4841] TS e3dbb92deec94038b08d065aa47a1bf1 (127.3.243.66:35763): tablet 61d88a7163ab4e9792269ad926891ba4 (table test-table [id=e3fe398264b54b3c98cb203ca4c69e8c]) successfully deleted
I20260619 18:47:08.694679 6381 ts_tablet_manager.cc:1918] T 61d88a7163ab4e9792269ad926891ba4 P b146718b0a3244aca8d5d949e5d8f13e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 18:47:08.694819 6381 log.cc:1196] T 61d88a7163ab4e9792269ad926891ba4 P b146718b0a3244aca8d5d949e5d8f13e: Deleting WAL directory at /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/minicluster-data/ts-0/wal/wals/61d88a7163ab4e9792269ad926891ba4
I20260619 18:47:08.695236 6381 ts_tablet_manager.cc:1939] T 61d88a7163ab4e9792269ad926891ba4 P b146718b0a3244aca8d5d949e5d8f13e: Deleting consensus metadata
I20260619 18:47:08.696319 5788 catalog_manager.cc:4841] TS b146718b0a3244aca8d5d949e5d8f13e (127.3.243.65:45537): tablet 61d88a7163ab4e9792269ad926891ba4 (table test-table [id=e3fe398264b54b3c98cb203ca4c69e8c]) successfully deleted
Jun 19 18:47:08 dist-test-slave-m9wg krb5kdc[5756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894828, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 18:47:08 dist-test-slave-m9wg krb5kdc[5756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894828, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.243.126@KRBTEST.COM
I20260619 18:47:08.723193 6391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:08.710063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48424 (local address 127.3.243.126:39867)
0619 18:47:08.710631 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:08.710637 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:08.710662 (+ 25us) server_negotiation.cc:407] Connection header received
0619 18:47:08.710809 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:08.710813 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:08.710884 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:08.710985 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:08.712351 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:08.713127 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:08.714350 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:08.714779 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:08.714887 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:08.718583 (+ 3696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:08.718589 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:08.718592 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:08.718624 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:08.721084 (+ 2460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:08.721689 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:08.721692 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:08.721695 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:08.721805 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:08.722077 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:08.722080 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:08.722082 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:08.722324 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:08.722492 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:08.722741 (+ 249us) server_negotiation.cc:299] Negotiation successful
0619 18:47:08.722916 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":265,"threads_started":1}
I20260619 18:47:08.726397 5799 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48424:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:47:08.726981 5799 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:47:08.732870 5938 tablet_service.cc:1467] Processing CreateTablet for tablet 363ddd7b29fd4b8b815e145c4f885491 (DEFAULT_TABLE table=test-table [id=d16cae253da942d185a45eaacbf4663c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:47:08.733017 6077 tablet_service.cc:1467] Processing CreateTablet for tablet 363ddd7b29fd4b8b815e145c4f885491 (DEFAULT_TABLE table=test-table [id=d16cae253da942d185a45eaacbf4663c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:47:08.733335 5938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 363ddd7b29fd4b8b815e145c4f885491. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:47:08.733385 6077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 363ddd7b29fd4b8b815e145c4f885491. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:47:08.733335 6215 tablet_service.cc:1467] Processing CreateTablet for tablet 363ddd7b29fd4b8b815e145c4f885491 (DEFAULT_TABLE table=test-table [id=d16cae253da942d185a45eaacbf4663c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:47:08.733734 6215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 363ddd7b29fd4b8b815e145c4f885491. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:47:08.738843 6392 tablet_bootstrap.cc:492] T 363ddd7b29fd4b8b815e145c4f885491 P efb278fc23954c9d86d8c54f68b6462e: Bootstrap starting.
I20260619 18:47:08.740414 6392 tablet_bootstrap.cc:654] T 363ddd7b29fd4b8b815e145c4f885491 P efb278fc23954c9d86d8c54f68b6462e: Neither blocks nor log segments found. Creating new log.
I20260619 18:47:08.740685 6393 tablet_bootstrap.cc:492] T 363ddd7b29fd4b8b815e145c4f885491 P b146718b0a3244aca8d5d949e5d8f13e: Bootstrap starting.
I20260619 18:47:08.741336 6394 tablet_bootstrap.cc:492] T 363ddd7b29fd4b8b815e145c4f885491 P e3dbb92deec94038b08d065aa47a1bf1: Bootstrap starting.
I20260619 18:47:08.742668 6393 tablet_bootstrap.cc:654] T 363ddd7b29fd4b8b815e145c4f885491 P b146718b0a3244aca8d5d949e5d8f13e: Neither blocks nor log segments found. Creating new log.
I20260619 18:47:08.742847 6392 tablet_bootstrap.cc:492] T 363ddd7b29fd4b8b815e145c4f885491 P efb278fc23954c9d86d8c54f68b6462e: No bootstrap required, opened a new log
I20260619 18:47:08.742996 6392 ts_tablet_manager.cc:1397] T 363ddd7b29fd4b8b815e145c4f885491 P efb278fc23954c9d86d8c54f68b6462e: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 18:47:08.743021 6394 tablet_bootstrap.cc:654] T 363ddd7b29fd4b8b815e145c4f885491 P e3dbb92deec94038b08d065aa47a1bf1: Neither blocks nor log segments found. Creating new log.
I20260619 18:47:08.743753 6392 raft_consensus.cc:348] T 363ddd7b29fd4b8b815e145c4f885491 P efb278fc23954c9d86d8c54f68b6462e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b146718b0a3244aca8d5d949e5d8f13e" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 45537 } } peers { permanent_uuid: "e3dbb92deec94038b08d065aa47a1bf1" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 35763 } } peers { permanent_uuid: "efb278fc23954c9d86d8c54f68b6462e" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 35465 } }
I20260619 18:47:08.743938 6392 raft_consensus.cc:374] T 363ddd7b29fd4b8b815e145c4f885491 P efb278fc23954c9d86d8c54f68b6462e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:47:08.744028 6392 raft_consensus.cc:729] T 363ddd7b29fd4b8b815e145c4f885491 P efb278fc23954c9d86d8c54f68b6462e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efb278fc23954c9d86d8c54f68b6462e, State: Initialized, Role: FOLLOWER
I20260619 18:47:08.744273 6392 consensus_queue.cc:260] T 363ddd7b29fd4b8b815e145c4f885491 P efb278fc23954c9d86d8c54f68b6462e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b146718b0a3244aca8d5d949e5d8f13e" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 45537 } } peers { permanent_uuid: "e3dbb92deec94038b08d065aa47a1bf1" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 35763 } } peers { permanent_uuid: "efb278fc23954c9d86d8c54f68b6462e" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 35465 } }
I20260619 18:47:08.744804 6393 tablet_bootstrap.cc:492] T 363ddd7b29fd4b8b815e145c4f885491 P b146718b0a3244aca8d5d949e5d8f13e: No bootstrap required, opened a new log
I20260619 18:47:08.744938 6392 ts_tablet_manager.cc:1428] T 363ddd7b29fd4b8b815e145c4f885491 P efb278fc23954c9d86d8c54f68b6462e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 18:47:08.745038 6393 ts_tablet_manager.cc:1397] T 363ddd7b29fd4b8b815e145c4f885491 P b146718b0a3244aca8d5d949e5d8f13e: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260619 18:47:08.746012 6393 raft_consensus.cc:348] T 363ddd7b29fd4b8b815e145c4f885491 P b146718b0a3244aca8d5d949e5d8f13e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b146718b0a3244aca8d5d949e5d8f13e" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 45537 } } peers { permanent_uuid: "e3dbb92deec94038b08d065aa47a1bf1" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 35763 } } peers { permanent_uuid: "efb278fc23954c9d86d8c54f68b6462e" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 35465 } }
I20260619 18:47:08.746287 6393 raft_consensus.cc:374] T 363ddd7b29fd4b8b815e145c4f885491 P b146718b0a3244aca8d5d949e5d8f13e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:47:08.746402 6393 raft_consensus.cc:729] T 363ddd7b29fd4b8b815e145c4f885491 P b146718b0a3244aca8d5d949e5d8f13e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b146718b0a3244aca8d5d949e5d8f13e, State: Initialized, Role: FOLLOWER
I20260619 18:47:08.746656 6394 tablet_bootstrap.cc:492] T 363ddd7b29fd4b8b815e145c4f885491 P e3dbb92deec94038b08d065aa47a1bf1: No bootstrap required, opened a new log
I20260619 18:47:08.746627 6393 consensus_queue.cc:260] T 363ddd7b29fd4b8b815e145c4f885491 P b146718b0a3244aca8d5d949e5d8f13e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b146718b0a3244aca8d5d949e5d8f13e" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 45537 } } peers { permanent_uuid: "e3dbb92deec94038b08d065aa47a1bf1" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 35763 } } peers { permanent_uuid: "efb278fc23954c9d86d8c54f68b6462e" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 35465 } }
I20260619 18:47:08.746785 6394 ts_tablet_manager.cc:1397] T 363ddd7b29fd4b8b815e145c4f885491 P e3dbb92deec94038b08d065aa47a1bf1: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260619 18:47:08.747582 6394 raft_consensus.cc:348] T 363ddd7b29fd4b8b815e145c4f885491 P e3dbb92deec94038b08d065aa47a1bf1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b146718b0a3244aca8d5d949e5d8f13e" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 45537 } } peers { permanent_uuid: "e3dbb92deec94038b08d065aa47a1bf1" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 35763 } } peers { permanent_uuid: "efb278fc23954c9d86d8c54f68b6462e" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 35465 } }
I20260619 18:47:08.747821 6394 raft_consensus.cc:374] T 363ddd7b29fd4b8b815e145c4f885491 P e3dbb92deec94038b08d065aa47a1bf1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:47:08.747921 6394 raft_consensus.cc:729] T 363ddd7b29fd4b8b815e145c4f885491 P e3dbb92deec94038b08d065aa47a1bf1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3dbb92deec94038b08d065aa47a1bf1, State: Initialized, Role: FOLLOWER
I20260619 18:47:08.748174 6394 consensus_queue.cc:260] T 363ddd7b29fd4b8b815e145c4f885491 P e3dbb92deec94038b08d065aa47a1bf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b146718b0a3244aca8d5d949e5d8f13e" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 45537 } } peers { permanent_uuid: "e3dbb92deec94038b08d065aa47a1bf1" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 35763 } } peers { permanent_uuid: "efb278fc23954c9d86d8c54f68b6462e" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 35465 } }
I20260619 18:47:08.749019 6394 ts_tablet_manager.cc:1428] T 363ddd7b29fd4b8b815e145c4f885491 P e3dbb92deec94038b08d065aa47a1bf1: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260619 18:47:08.749748 6393 ts_tablet_manager.cc:1428] T 363ddd7b29fd4b8b815e145c4f885491 P b146718b0a3244aca8d5d949e5d8f13e: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260619 18:47:08.764797 6376 raft_consensus.cc:482] T 363ddd7b29fd4b8b815e145c4f885491 P efb278fc23954c9d86d8c54f68b6462e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:47:08.764964 6376 raft_consensus.cc:504] T 363ddd7b29fd4b8b815e145c4f885491 P efb278fc23954c9d86d8c54f68b6462e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b146718b0a3244aca8d5d949e5d8f13e" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 45537 } } peers { permanent_uuid: "e3dbb92deec94038b08d065aa47a1bf1" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 35763 } } peers { permanent_uuid: "efb278fc23954c9d86d8c54f68b6462e" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 35465 } }
I20260619 18:47:08.765522 6376 leader_election.cc:288] T 363ddd7b29fd4b8b815e145c4f885491 P efb278fc23954c9d86d8c54f68b6462e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b146718b0a3244aca8d5d949e5d8f13e (127.3.243.65:45537), e3dbb92deec94038b08d065aa47a1bf1 (127.3.243.66:35763)
I20260619 18:47:08.765964 5958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "363ddd7b29fd4b8b815e145c4f885491" candidate_uuid: "efb278fc23954c9d86d8c54f68b6462e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b146718b0a3244aca8d5d949e5d8f13e" is_pre_election: true
I20260619 18:47:08.766002 6097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "363ddd7b29fd4b8b815e145c4f885491" candidate_uuid: "efb278fc23954c9d86d8c54f68b6462e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3dbb92deec94038b08d065aa47a1bf1" is_pre_election: true
I20260619 18:47:08.766197 5958 raft_consensus.cc:2454] T 363ddd7b29fd4b8b815e145c4f885491 P b146718b0a3244aca8d5d949e5d8f13e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efb278fc23954c9d86d8c54f68b6462e in term 0.
I20260619 18:47:08.766242 6097 raft_consensus.cc:2454] T 363ddd7b29fd4b8b815e145c4f885491 P e3dbb92deec94038b08d065aa47a1bf1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efb278fc23954c9d86d8c54f68b6462e in term 0.
I20260619 18:47:08.766631 6170 leader_election.cc:302] T 363ddd7b29fd4b8b815e145c4f885491 P efb278fc23954c9d86d8c54f68b6462e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b146718b0a3244aca8d5d949e5d8f13e, efb278fc23954c9d86d8c54f68b6462e; no voters:
I20260619 18:47:08.766886 6376 raft_consensus.cc:2781] T 363ddd7b29fd4b8b815e145c4f885491 P efb278fc23954c9d86d8c54f68b6462e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:47:08.767024 6376 raft_consensus.cc:482] T 363ddd7b29fd4b8b815e145c4f885491 P efb278fc23954c9d86d8c54f68b6462e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:47:08.767127 6376 raft_consensus.cc:3037] T 363ddd7b29fd4b8b815e145c4f885491 P efb278fc23954c9d86d8c54f68b6462e [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:47:08.769201 6376 raft_consensus.cc:504] T 363ddd7b29fd4b8b815e145c4f885491 P efb278fc23954c9d86d8c54f68b6462e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b146718b0a3244aca8d5d949e5d8f13e" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 45537 } } peers { permanent_uuid: "e3dbb92deec94038b08d065aa47a1bf1" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 35763 } } peers { permanent_uuid: "efb278fc23954c9d86d8c54f68b6462e" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 35465 } }
I20260619 18:47:08.769677 6376 leader_election.cc:288] T 363ddd7b29fd4b8b815e145c4f885491 P efb278fc23954c9d86d8c54f68b6462e [CANDIDATE]: Term 1 election: Requested vote from peers b146718b0a3244aca8d5d949e5d8f13e (127.3.243.65:45537), e3dbb92deec94038b08d065aa47a1bf1 (127.3.243.66:35763)
I20260619 18:47:08.770123 6097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "363ddd7b29fd4b8b815e145c4f885491" candidate_uuid: "efb278fc23954c9d86d8c54f68b6462e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3dbb92deec94038b08d065aa47a1bf1"
I20260619 18:47:08.770121 5958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "363ddd7b29fd4b8b815e145c4f885491" candidate_uuid: "efb278fc23954c9d86d8c54f68b6462e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b146718b0a3244aca8d5d949e5d8f13e"
I20260619 18:47:08.770334 6097 raft_consensus.cc:3037] T 363ddd7b29fd4b8b815e145c4f885491 P e3dbb92deec94038b08d065aa47a1bf1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:47:08.770334 5958 raft_consensus.cc:3037] T 363ddd7b29fd4b8b815e145c4f885491 P b146718b0a3244aca8d5d949e5d8f13e [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:47:08.773159 6097 raft_consensus.cc:2454] T 363ddd7b29fd4b8b815e145c4f885491 P e3dbb92deec94038b08d065aa47a1bf1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efb278fc23954c9d86d8c54f68b6462e in term 1.
I20260619 18:47:08.773159 5958 raft_consensus.cc:2454] T 363ddd7b29fd4b8b815e145c4f885491 P b146718b0a3244aca8d5d949e5d8f13e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efb278fc23954c9d86d8c54f68b6462e in term 1.
I20260619 18:47:08.773569 6170 leader_election.cc:302] T 363ddd7b29fd4b8b815e145c4f885491 P efb278fc23954c9d86d8c54f68b6462e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: b146718b0a3244aca8d5d949e5d8f13e, e3dbb92deec94038b08d065aa47a1bf1, efb278fc23954c9d86d8c54f68b6462e; no voters:
I20260619 18:47:08.773859 6376 raft_consensus.cc:2781] T 363ddd7b29fd4b8b815e145c4f885491 P efb278fc23954c9d86d8c54f68b6462e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:47:08.773989 6376 raft_consensus.cc:686] T 363ddd7b29fd4b8b815e145c4f885491 P efb278fc23954c9d86d8c54f68b6462e [term 1 LEADER]: Becoming Leader. State: Replica: efb278fc23954c9d86d8c54f68b6462e, State: Running, Role: LEADER
I20260619 18:47:08.774240 6376 consensus_queue.cc:237] T 363ddd7b29fd4b8b815e145c4f885491 P efb278fc23954c9d86d8c54f68b6462e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b146718b0a3244aca8d5d949e5d8f13e" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 45537 } } peers { permanent_uuid: "e3dbb92deec94038b08d065aa47a1bf1" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 35763 } } peers { permanent_uuid: "efb278fc23954c9d86d8c54f68b6462e" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 35465 } }
I20260619 18:47:08.776681 5800 catalog_manager.cc:5495] T 363ddd7b29fd4b8b815e145c4f885491 P efb278fc23954c9d86d8c54f68b6462e reported cstate change: term changed from 0 to 1, leader changed from <none> to efb278fc23954c9d86d8c54f68b6462e (127.3.243.67). New cstate: current_term: 1 leader_uuid: "efb278fc23954c9d86d8c54f68b6462e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b146718b0a3244aca8d5d949e5d8f13e" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 45537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3dbb92deec94038b08d065aa47a1bf1" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 35763 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efb278fc23954c9d86d8c54f68b6462e" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 35465 } health_report { overall_health: HEALTHY } } }
I20260619 18:47:08.803020 6402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:08.797986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39998 (local address 127.3.243.67:35465)
0619 18:47:08.798557 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:08.798561 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:08.798577 (+ 16us) server_negotiation.cc:407] Connection header received
0619 18:47:08.798624 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:08.798628 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:08.798685 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:08.798764 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:47:08.799159 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:08.799972 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:08.801311 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:08.801740 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:08.801845 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:08.802304 (+ 459us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:47:08.802406 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:47:08.802490 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:08.802639 (+ 149us) server_negotiation.cc:299] Negotiation successful
0619 18:47:08.802714 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":271,"threads_started":1}
I20260619 18:47:08.810986 5958 raft_consensus.cc:1261] T 363ddd7b29fd4b8b815e145c4f885491 P b146718b0a3244aca8d5d949e5d8f13e [term 1 FOLLOWER]: Refusing update from remote peer efb278fc23954c9d86d8c54f68b6462e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:47:08.811160 6097 raft_consensus.cc:1261] T 363ddd7b29fd4b8b815e145c4f885491 P e3dbb92deec94038b08d065aa47a1bf1 [term 1 FOLLOWER]: Refusing update from remote peer efb278fc23954c9d86d8c54f68b6462e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:47:08.811502 6364 consensus_queue.cc:1035] T 363ddd7b29fd4b8b815e145c4f885491 P efb278fc23954c9d86d8c54f68b6462e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b146718b0a3244aca8d5d949e5d8f13e" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 45537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:47:08.811832 6361 consensus_queue.cc:1035] T 363ddd7b29fd4b8b815e145c4f885491 P efb278fc23954c9d86d8c54f68b6462e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3dbb92deec94038b08d065aa47a1bf1" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 35763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:47:08.850608 5800 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48424:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:47:08.850823 5800 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48424:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:47:08.853323 5800 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7647451422aa4ed39c38da3831664f1b: Sending DeleteTablet for 3 replicas of tablet 363ddd7b29fd4b8b815e145c4f885491
I20260619 18:47:08.854034 5938 tablet_service.cc:1514] Processing DeleteTablet for tablet 363ddd7b29fd4b8b815e145c4f885491 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:47:08 UTC) from {username='kudu', principal='kudu/127.3.243.126@KRBTEST.COM'} at 127.0.0.1:50528
I20260619 18:47:08.854094 6216 tablet_service.cc:1514] Processing DeleteTablet for tablet 363ddd7b29fd4b8b815e145c4f885491 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:47:08 UTC) from {username='kudu', principal='kudu/127.3.243.126@KRBTEST.COM'} at 127.0.0.1:39980
I20260619 18:47:08.854144 6077 tablet_service.cc:1514] Processing DeleteTablet for tablet 363ddd7b29fd4b8b815e145c4f885491 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:47:08 UTC) from {username='kudu', principal='kudu/127.3.243.126@KRBTEST.COM'} at 127.0.0.1:32794
I20260619 18:47:08.854369 6382 tablet_replica.cc:330] T 363ddd7b29fd4b8b815e145c4f885491 P e3dbb92deec94038b08d065aa47a1bf1: stopping tablet replica
I20260619 18:47:08.854377 6383 tablet_replica.cc:330] T 363ddd7b29fd4b8b815e145c4f885491 P efb278fc23954c9d86d8c54f68b6462e: stopping tablet replica
I20260619 18:47:08.854519 6382 raft_consensus.cc:2229] T 363ddd7b29fd4b8b815e145c4f885491 P e3dbb92deec94038b08d065aa47a1bf1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:47:08.854580 6383 raft_consensus.cc:2229] T 363ddd7b29fd4b8b815e145c4f885491 P efb278fc23954c9d86d8c54f68b6462e [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:47:08.854665 6382 raft_consensus.cc:2258] T 363ddd7b29fd4b8b815e145c4f885491 P e3dbb92deec94038b08d065aa47a1bf1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:47:08.854714 6381 tablet_replica.cc:330] T 363ddd7b29fd4b8b815e145c4f885491 P b146718b0a3244aca8d5d949e5d8f13e: stopping tablet replica
I20260619 18:47:08.854866 6383 raft_consensus.cc:2258] T 363ddd7b29fd4b8b815e145c4f885491 P efb278fc23954c9d86d8c54f68b6462e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:47:08.854897 6381 raft_consensus.cc:2229] T 363ddd7b29fd4b8b815e145c4f885491 P b146718b0a3244aca8d5d949e5d8f13e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:47:08.855058 6381 raft_consensus.cc:2258] T 363ddd7b29fd4b8b815e145c4f885491 P b146718b0a3244aca8d5d949e5d8f13e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:47:08.855317 5800 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48424:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 18:47:08.855851 6383 ts_tablet_manager.cc:1905] T 363ddd7b29fd4b8b815e145c4f885491 P efb278fc23954c9d86d8c54f68b6462e: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260619 18:47:08.855957 5800 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:47:08.856091 6381 ts_tablet_manager.cc:1905] T 363ddd7b29fd4b8b815e145c4f885491 P b146718b0a3244aca8d5d949e5d8f13e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:47:08.856323 6382 ts_tablet_manager.cc:1905] T 363ddd7b29fd4b8b815e145c4f885491 P e3dbb92deec94038b08d065aa47a1bf1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:47:08.860687 6381 ts_tablet_manager.cc:1918] T 363ddd7b29fd4b8b815e145c4f885491 P b146718b0a3244aca8d5d949e5d8f13e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 18:47:08.860828 6381 log.cc:1196] T 363ddd7b29fd4b8b815e145c4f885491 P b146718b0a3244aca8d5d949e5d8f13e: Deleting WAL directory at /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/minicluster-data/ts-0/wal/wals/363ddd7b29fd4b8b815e145c4f885491
I20260619 18:47:08.861195 6381 ts_tablet_manager.cc:1939] T 363ddd7b29fd4b8b815e145c4f885491 P b146718b0a3244aca8d5d949e5d8f13e: Deleting consensus metadata
I20260619 18:47:08.862172 5788 catalog_manager.cc:4841] TS b146718b0a3244aca8d5d949e5d8f13e (127.3.243.65:45537): tablet 363ddd7b29fd4b8b815e145c4f885491 (table test-table [id=d16cae253da942d185a45eaacbf4663c]) successfully deleted
I20260619 18:47:08.862277 6383 ts_tablet_manager.cc:1918] T 363ddd7b29fd4b8b815e145c4f885491 P efb278fc23954c9d86d8c54f68b6462e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 18:47:08.862360 6382 ts_tablet_manager.cc:1918] T 363ddd7b29fd4b8b815e145c4f885491 P e3dbb92deec94038b08d065aa47a1bf1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 18:47:08.862414 6383 log.cc:1196] T 363ddd7b29fd4b8b815e145c4f885491 P efb278fc23954c9d86d8c54f68b6462e: Deleting WAL directory at /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/minicluster-data/ts-2/wal/wals/363ddd7b29fd4b8b815e145c4f885491
I20260619 18:47:08.862478 6382 log.cc:1196] T 363ddd7b29fd4b8b815e145c4f885491 P e3dbb92deec94038b08d065aa47a1bf1: Deleting WAL directory at /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/minicluster-data/ts-1/wal/wals/363ddd7b29fd4b8b815e145c4f885491
I20260619 18:47:08.862799 6383 ts_tablet_manager.cc:1939] T 363ddd7b29fd4b8b815e145c4f885491 P efb278fc23954c9d86d8c54f68b6462e: Deleting consensus metadata
I20260619 18:47:08.862833 6382 ts_tablet_manager.cc:1939] T 363ddd7b29fd4b8b815e145c4f885491 P e3dbb92deec94038b08d065aa47a1bf1: Deleting consensus metadata
I20260619 18:47:08.863688 5787 catalog_manager.cc:4841] TS efb278fc23954c9d86d8c54f68b6462e (127.3.243.67:35465): tablet 363ddd7b29fd4b8b815e145c4f885491 (table test-table [id=d16cae253da942d185a45eaacbf4663c]) successfully deleted
I20260619 18:47:08.864249 5787 catalog_manager.cc:4841] TS e3dbb92deec94038b08d065aa47a1bf1 (127.3.243.66:35763): tablet 363ddd7b29fd4b8b815e145c4f885491 (table test-table [id=d16cae253da942d185a45eaacbf4663c]) successfully deleted
I20260619 18:47:08.866084 5938 tablet_service.cc:1467] Processing CreateTablet for tablet 53352630a3c54327b176b5c595826738 (DEFAULT_TABLE table=test-table [id=f7b4036f385d446084d100ab75a61e53]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:47:08.866209 6077 tablet_service.cc:1467] Processing CreateTablet for tablet 53352630a3c54327b176b5c595826738 (DEFAULT_TABLE table=test-table [id=f7b4036f385d446084d100ab75a61e53]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:47:08.866427 5938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53352630a3c54327b176b5c595826738. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:47:08.866587 6077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53352630a3c54327b176b5c595826738. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:47:08.870386 6393 tablet_bootstrap.cc:492] T 53352630a3c54327b176b5c595826738 P b146718b0a3244aca8d5d949e5d8f13e: Bootstrap starting.
I20260619 18:47:08.870565 6394 tablet_bootstrap.cc:492] T 53352630a3c54327b176b5c595826738 P e3dbb92deec94038b08d065aa47a1bf1: Bootstrap starting.
I20260619 18:47:08.871094 6216 tablet_service.cc:1467] Processing CreateTablet for tablet 53352630a3c54327b176b5c595826738 (DEFAULT_TABLE table=test-table [id=f7b4036f385d446084d100ab75a61e53]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:47:08.871455 6216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53352630a3c54327b176b5c595826738. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:47:08.871780 6393 tablet_bootstrap.cc:654] T 53352630a3c54327b176b5c595826738 P b146718b0a3244aca8d5d949e5d8f13e: Neither blocks nor log segments found. Creating new log.
I20260619 18:47:08.871992 6394 tablet_bootstrap.cc:654] T 53352630a3c54327b176b5c595826738 P e3dbb92deec94038b08d065aa47a1bf1: Neither blocks nor log segments found. Creating new log.
I20260619 18:47:08.873797 6393 tablet_bootstrap.cc:492] T 53352630a3c54327b176b5c595826738 P b146718b0a3244aca8d5d949e5d8f13e: No bootstrap required, opened a new log
I20260619 18:47:08.873958 6393 ts_tablet_manager.cc:1397] T 53352630a3c54327b176b5c595826738 P b146718b0a3244aca8d5d949e5d8f13e: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260619 18:47:08.874150 6394 tablet_bootstrap.cc:492] T 53352630a3c54327b176b5c595826738 P e3dbb92deec94038b08d065aa47a1bf1: No bootstrap required, opened a new log
I20260619 18:47:08.874294 6394 ts_tablet_manager.cc:1397] T 53352630a3c54327b176b5c595826738 P e3dbb92deec94038b08d065aa47a1bf1: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260619 18:47:08.874564 6393 raft_consensus.cc:348] T 53352630a3c54327b176b5c595826738 P b146718b0a3244aca8d5d949e5d8f13e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b146718b0a3244aca8d5d949e5d8f13e" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 45537 } } peers { permanent_uuid: "e3dbb92deec94038b08d065aa47a1bf1" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 35763 } } peers { permanent_uuid: "efb278fc23954c9d86d8c54f68b6462e" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 35465 } }
I20260619 18:47:08.874751 6393 raft_consensus.cc:374] T 53352630a3c54327b176b5c595826738 P b146718b0a3244aca8d5d949e5d8f13e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:47:08.874826 6393 raft_consensus.cc:729] T 53352630a3c54327b176b5c595826738 P b146718b0a3244aca8d5d949e5d8f13e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b146718b0a3244aca8d5d949e5d8f13e, State: Initialized, Role: FOLLOWER
I20260619 18:47:08.875048 6393 consensus_queue.cc:260] T 53352630a3c54327b176b5c595826738 P b146718b0a3244aca8d5d949e5d8f13e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b146718b0a3244aca8d5d949e5d8f13e" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 45537 } } peers { permanent_uuid: "e3dbb92deec94038b08d065aa47a1bf1" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 35763 } } peers { permanent_uuid: "efb278fc23954c9d86d8c54f68b6462e" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 35465 } }
I20260619 18:47:08.874970 6394 raft_consensus.cc:348] T 53352630a3c54327b176b5c595826738 P e3dbb92deec94038b08d065aa47a1bf1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b146718b0a3244aca8d5d949e5d8f13e" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 45537 } } peers { permanent_uuid: "e3dbb92deec94038b08d065aa47a1bf1" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 35763 } } peers { permanent_uuid: "efb278fc23954c9d86d8c54f68b6462e" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 35465 } }
I20260619 18:47:08.875221 6394 raft_consensus.cc:374] T 53352630a3c54327b176b5c595826738 P e3dbb92deec94038b08d065aa47a1bf1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:47:08.875312 6394 raft_consensus.cc:729] T 53352630a3c54327b176b5c595826738 P e3dbb92deec94038b08d065aa47a1bf1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3dbb92deec94038b08d065aa47a1bf1, State: Initialized, Role: FOLLOWER
I20260619 18:47:08.875483 6393 ts_tablet_manager.cc:1428] T 53352630a3c54327b176b5c595826738 P b146718b0a3244aca8d5d949e5d8f13e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260619 18:47:08.875550 6394 consensus_queue.cc:260] T 53352630a3c54327b176b5c595826738 P e3dbb92deec94038b08d065aa47a1bf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b146718b0a3244aca8d5d949e5d8f13e" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 45537 } } peers { permanent_uuid: "e3dbb92deec94038b08d065aa47a1bf1" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 35763 } } peers { permanent_uuid: "efb278fc23954c9d86d8c54f68b6462e" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 35465 } }
I20260619 18:47:08.876019 6394 ts_tablet_manager.cc:1428] T 53352630a3c54327b176b5c595826738 P e3dbb92deec94038b08d065aa47a1bf1: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260619 18:47:08.878472 6392 tablet_bootstrap.cc:492] T 53352630a3c54327b176b5c595826738 P efb278fc23954c9d86d8c54f68b6462e: Bootstrap starting.
I20260619 18:47:08.879803 6392 tablet_bootstrap.cc:654] T 53352630a3c54327b176b5c595826738 P efb278fc23954c9d86d8c54f68b6462e: Neither blocks nor log segments found. Creating new log.
I20260619 18:47:08.881460 6392 tablet_bootstrap.cc:492] T 53352630a3c54327b176b5c595826738 P efb278fc23954c9d86d8c54f68b6462e: No bootstrap required, opened a new log
I20260619 18:47:08.881600 6392 ts_tablet_manager.cc:1397] T 53352630a3c54327b176b5c595826738 P efb278fc23954c9d86d8c54f68b6462e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 18:47:08.882114 6392 raft_consensus.cc:348] T 53352630a3c54327b176b5c595826738 P efb278fc23954c9d86d8c54f68b6462e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b146718b0a3244aca8d5d949e5d8f13e" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 45537 } } peers { permanent_uuid: "e3dbb92deec94038b08d065aa47a1bf1" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 35763 } } peers { permanent_uuid: "efb278fc23954c9d86d8c54f68b6462e" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 35465 } }
I20260619 18:47:08.882298 6392 raft_consensus.cc:374] T 53352630a3c54327b176b5c595826738 P efb278fc23954c9d86d8c54f68b6462e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:47:08.882370 6392 raft_consensus.cc:729] T 53352630a3c54327b176b5c595826738 P efb278fc23954c9d86d8c54f68b6462e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efb278fc23954c9d86d8c54f68b6462e, State: Initialized, Role: FOLLOWER
I20260619 18:47:08.882583 6392 consensus_queue.cc:260] T 53352630a3c54327b176b5c595826738 P efb278fc23954c9d86d8c54f68b6462e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b146718b0a3244aca8d5d949e5d8f13e" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 45537 } } peers { permanent_uuid: "e3dbb92deec94038b08d065aa47a1bf1" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 35763 } } peers { permanent_uuid: "efb278fc23954c9d86d8c54f68b6462e" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 35465 } }
I20260619 18:47:08.882967 6392 ts_tablet_manager.cc:1428] T 53352630a3c54327b176b5c595826738 P efb278fc23954c9d86d8c54f68b6462e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 18:47:08.888514 6399 raft_consensus.cc:482] T 53352630a3c54327b176b5c595826738 P e3dbb92deec94038b08d065aa47a1bf1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:47:08.888666 6399 raft_consensus.cc:504] T 53352630a3c54327b176b5c595826738 P e3dbb92deec94038b08d065aa47a1bf1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b146718b0a3244aca8d5d949e5d8f13e" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 45537 } } peers { permanent_uuid: "e3dbb92deec94038b08d065aa47a1bf1" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 35763 } } peers { permanent_uuid: "efb278fc23954c9d86d8c54f68b6462e" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 35465 } }
I20260619 18:47:08.890137 6399 leader_election.cc:288] T 53352630a3c54327b176b5c595826738 P e3dbb92deec94038b08d065aa47a1bf1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b146718b0a3244aca8d5d949e5d8f13e (127.3.243.65:45537), efb278fc23954c9d86d8c54f68b6462e (127.3.243.67:35465)
I20260619 18:47:08.896775 6402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:08.890527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.66:47015 (local address 127.3.243.67:35465)
0619 18:47:08.891317 (+ 790us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:08.891324 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:08.891345 (+ 21us) server_negotiation.cc:407] Connection header received
0619 18:47:08.891409 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:08.891415 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:08.891493 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:08.891575 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:47:08.892175 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:08.892905 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:08.894899 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:08.895828 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:08.895958 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:08.896010 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:08.896415 (+ 405us) server_negotiation.cc:299] Negotiation successful
0619 18:47:08.896551 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":663}
I20260619 18:47:08.896839 6409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:08.890422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.67:35465 (local address 127.3.243.66:47015)
0619 18:47:08.891013 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:08.891033 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:08.891176 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:08.891612 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:08.891617 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:08.891651 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:47:08.891993 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:08.892011 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:08.893030 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:08.893035 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:08.894753 (+ 1718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:08.894768 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:08.896021 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:08.896027 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:08.896231 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:08.896278 (+ 47us) client_negotiation.cc:769] Sending connection context
0619 18:47:08.896363 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 18:47:08.896506 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"mutex_wait_us":53,"thread_start_us":301,"threads_started":1}
I20260619 18:47:08.897169 6410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:08.890549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.66:40507 (local address 127.3.243.65:45537)
0619 18:47:08.891654 (+ 1105us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:08.891659 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:08.891680 (+ 21us) server_negotiation.cc:407] Connection header received
0619 18:47:08.891736 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:08.891740 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:08.891799 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:08.891889 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:47:08.892459 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:08.893207 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:08.895286 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:08.896171 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:08.896364 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:08.896440 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:08.896741 (+ 301us) server_negotiation.cc:299] Negotiation successful
0619 18:47:08.896872 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":989,"thread_start_us":856,"threads_started":1}
I20260619 18:47:08.897450 6411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:08.890553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.65:45537 (local address 127.3.243.66:40507)
0619 18:47:08.891297 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:08.891313 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:08.891409 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:08.891930 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:08.891934 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:08.891958 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:47:08.892296 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:08.892305 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:08.893332 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:08.893337 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:08.895160 (+ 1823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:08.895174 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:08.896380 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:08.896387 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:08.896568 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:08.896605 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 18:47:08.896689 (+ 84us) client_negotiation.cc:240] Negotiation successful
0619 18:47:08.896816 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"mutex_wait_us":44,"thread_start_us":456,"threads_started":1}
I20260619 18:47:08.897691 6236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53352630a3c54327b176b5c595826738" candidate_uuid: "e3dbb92deec94038b08d065aa47a1bf1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efb278fc23954c9d86d8c54f68b6462e" is_pre_election: true
I20260619 18:47:08.898030 5958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53352630a3c54327b176b5c595826738" candidate_uuid: "e3dbb92deec94038b08d065aa47a1bf1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b146718b0a3244aca8d5d949e5d8f13e" is_pre_election: true
I20260619 18:47:08.898100 6236 raft_consensus.cc:2454] T 53352630a3c54327b176b5c595826738 P efb278fc23954c9d86d8c54f68b6462e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3dbb92deec94038b08d065aa47a1bf1 in term 0.
I20260619 18:47:08.898286 5958 raft_consensus.cc:2454] T 53352630a3c54327b176b5c595826738 P b146718b0a3244aca8d5d949e5d8f13e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3dbb92deec94038b08d065aa47a1bf1 in term 0.
I20260619 18:47:08.898705 6030 leader_election.cc:302] T 53352630a3c54327b176b5c595826738 P e3dbb92deec94038b08d065aa47a1bf1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: b146718b0a3244aca8d5d949e5d8f13e, e3dbb92deec94038b08d065aa47a1bf1, efb278fc23954c9d86d8c54f68b6462e; no voters:
I20260619 18:47:08.899068 6399 raft_consensus.cc:2781] T 53352630a3c54327b176b5c595826738 P e3dbb92deec94038b08d065aa47a1bf1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:47:08.899183 6399 raft_consensus.cc:482] T 53352630a3c54327b176b5c595826738 P e3dbb92deec94038b08d065aa47a1bf1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:47:08.899282 6399 raft_consensus.cc:3037] T 53352630a3c54327b176b5c595826738 P e3dbb92deec94038b08d065aa47a1bf1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:47:08.901453 6399 raft_consensus.cc:504] T 53352630a3c54327b176b5c595826738 P e3dbb92deec94038b08d065aa47a1bf1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b146718b0a3244aca8d5d949e5d8f13e" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 45537 } } peers { permanent_uuid: "e3dbb92deec94038b08d065aa47a1bf1" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 35763 } } peers { permanent_uuid: "efb278fc23954c9d86d8c54f68b6462e" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 35465 } }
I20260619 18:47:08.902033 6399 leader_election.cc:288] T 53352630a3c54327b176b5c595826738 P e3dbb92deec94038b08d065aa47a1bf1 [CANDIDATE]: Term 1 election: Requested vote from peers b146718b0a3244aca8d5d949e5d8f13e (127.3.243.65:45537), efb278fc23954c9d86d8c54f68b6462e (127.3.243.67:35465)
I20260619 18:47:08.902360 5958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53352630a3c54327b176b5c595826738" candidate_uuid: "e3dbb92deec94038b08d065aa47a1bf1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b146718b0a3244aca8d5d949e5d8f13e"
I20260619 18:47:08.902484 6236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53352630a3c54327b176b5c595826738" candidate_uuid: "e3dbb92deec94038b08d065aa47a1bf1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efb278fc23954c9d86d8c54f68b6462e"
I20260619 18:47:08.902591 5958 raft_consensus.cc:3037] T 53352630a3c54327b176b5c595826738 P b146718b0a3244aca8d5d949e5d8f13e [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:47:08.902673 6236 raft_consensus.cc:3037] T 53352630a3c54327b176b5c595826738 P efb278fc23954c9d86d8c54f68b6462e [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:47:08.905512 5958 raft_consensus.cc:2454] T 53352630a3c54327b176b5c595826738 P b146718b0a3244aca8d5d949e5d8f13e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3dbb92deec94038b08d065aa47a1bf1 in term 1.
I20260619 18:47:08.905540 6236 raft_consensus.cc:2454] T 53352630a3c54327b176b5c595826738 P efb278fc23954c9d86d8c54f68b6462e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3dbb92deec94038b08d065aa47a1bf1 in term 1.
I20260619 18:47:08.905864 6031 leader_election.cc:302] T 53352630a3c54327b176b5c595826738 P e3dbb92deec94038b08d065aa47a1bf1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b146718b0a3244aca8d5d949e5d8f13e, e3dbb92deec94038b08d065aa47a1bf1; no voters:
I20260619 18:47:08.906116 6399 raft_consensus.cc:2781] T 53352630a3c54327b176b5c595826738 P e3dbb92deec94038b08d065aa47a1bf1 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:47:08.906546 6399 raft_consensus.cc:686] T 53352630a3c54327b176b5c595826738 P e3dbb92deec94038b08d065aa47a1bf1 [term 1 LEADER]: Becoming Leader. State: Replica: e3dbb92deec94038b08d065aa47a1bf1, State: Running, Role: LEADER
I20260619 18:47:08.906942 6399 consensus_queue.cc:237] T 53352630a3c54327b176b5c595826738 P e3dbb92deec94038b08d065aa47a1bf1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b146718b0a3244aca8d5d949e5d8f13e" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 45537 } } peers { permanent_uuid: "e3dbb92deec94038b08d065aa47a1bf1" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 35763 } } peers { permanent_uuid: "efb278fc23954c9d86d8c54f68b6462e" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 35465 } }
I20260619 18:47:08.910089 5800 catalog_manager.cc:5495] T 53352630a3c54327b176b5c595826738 P e3dbb92deec94038b08d065aa47a1bf1 reported cstate change: term changed from 0 to 1, leader changed from <none> to e3dbb92deec94038b08d065aa47a1bf1 (127.3.243.66). New cstate: current_term: 1 leader_uuid: "e3dbb92deec94038b08d065aa47a1bf1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b146718b0a3244aca8d5d949e5d8f13e" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 45537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3dbb92deec94038b08d065aa47a1bf1" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 35763 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "efb278fc23954c9d86d8c54f68b6462e" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 35465 } health_report { overall_health: UNKNOWN } } }
I20260619 18:47:08.935734 6414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:08.930517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32808 (local address 127.3.243.66:35763)
0619 18:47:08.931070 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:08.931076 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:08.931098 (+ 22us) server_negotiation.cc:407] Connection header received
0619 18:47:08.931216 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:08.931221 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:08.931305 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:08.931434 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:47:08.931776 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:08.932543 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:08.933755 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:08.934166 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:08.934253 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:08.934612 (+ 359us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:47:08.935206 (+ 594us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:47:08.935310 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:08.935414 (+ 104us) server_negotiation.cc:299] Negotiation successful
0619 18:47:08.935477 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":285,"threads_started":1}
I20260619 18:47:08.949947 6402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:08.944679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.66:48439 (local address 127.3.243.67:35465)
0619 18:47:08.944823 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:08.944827 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:08.945335 (+ 508us) server_negotiation.cc:407] Connection header received
0619 18:47:08.945535 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:08.945539 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:08.945601 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:08.945669 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:47:08.946129 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:08.946886 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:08.948290 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:08.948695 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:08.948804 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:08.949228 (+ 424us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:47:08.949342 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:47:08.949414 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:08.949559 (+ 145us) server_negotiation.cc:299] Negotiation successful
0619 18:47:08.949636 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 18:47:08.950008 6417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:08.944637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.67:35465 (local address 127.3.243.66:48439)
0619 18:47:08.945242 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:08.945257 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:08.945387 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:08.945695 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:08.945700 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:08.945711 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 18:47:08.945998 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:08.946006 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:08.947037 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:08.947043 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:08.948170 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:08.948186 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:08.948874 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:08.948881 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:08.949014 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:08.949063 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 18:47:08.949440 (+ 377us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 18:47:08.949451 (+ 11us) client_negotiation.cc:769] Sending connection context
0619 18:47:08.949522 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 18:47:08.949668 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":320,"threads_started":1}
I20260619 18:47:08.957397 5958 raft_consensus.cc:1261] T 53352630a3c54327b176b5c595826738 P b146718b0a3244aca8d5d949e5d8f13e [term 1 FOLLOWER]: Refusing update from remote peer e3dbb92deec94038b08d065aa47a1bf1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:47:08.957660 6236 raft_consensus.cc:1261] T 53352630a3c54327b176b5c595826738 P efb278fc23954c9d86d8c54f68b6462e [term 1 FOLLOWER]: Refusing update from remote peer e3dbb92deec94038b08d065aa47a1bf1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:47:08.957943 6412 consensus_queue.cc:1035] T 53352630a3c54327b176b5c595826738 P e3dbb92deec94038b08d065aa47a1bf1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b146718b0a3244aca8d5d949e5d8f13e" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 45537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:47:08.958294 6399 consensus_queue.cc:1035] T 53352630a3c54327b176b5c595826738 P e3dbb92deec94038b08d065aa47a1bf1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efb278fc23954c9d86d8c54f68b6462e" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 35465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:47:08.983418 6422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:08.978165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.66:35763 (local address 127.3.243.67:52545)
0619 18:47:08.978690 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:08.978705 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:08.978828 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:08.979115 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:08.979120 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:08.979132 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 18:47:08.979502 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:08.979516 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:08.980598 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:08.980603 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:08.981839 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:08.981852 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:08.982497 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:08.982501 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:08.982609 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:08.982643 (+ 34us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 18:47:08.982905 (+ 262us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 18:47:08.982912 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 18:47:08.982972 (+ 60us) client_negotiation.cc:240] Negotiation successful
0619 18:47:08.983068 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":265,"threads_started":1}
I20260619 18:47:08.983436 6414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:08.978252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.67:52545 (local address 127.3.243.66:35763)
0619 18:47:08.978416 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:08.978422 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:08.978779 (+ 357us) server_negotiation.cc:407] Connection header received
0619 18:47:08.978955 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:08.978961 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:08.979024 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:08.979090 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:47:08.979689 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:08.980485 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:08.981966 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:08.982368 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:08.982457 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:08.982754 (+ 297us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:47:08.982843 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:47:08.982906 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:08.983126 (+ 220us) server_negotiation.cc:299] Negotiation successful
0619 18:47:08.983185 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260619 18:47:08.984651 6365 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 18:47:10.583086 5800 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48424:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:47:10.583333 5800 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48424:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:47:10.587051 5800 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7647451422aa4ed39c38da3831664f1b: Sending DeleteTablet for 3 replicas of tablet 53352630a3c54327b176b5c595826738
I20260619 18:47:10.587729 5938 tablet_service.cc:1514] Processing DeleteTablet for tablet 53352630a3c54327b176b5c595826738 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:47:10 UTC) from {username='kudu', principal='kudu/127.3.243.126@KRBTEST.COM'} at 127.0.0.1:50528
I20260619 18:47:10.587790 6077 tablet_service.cc:1514] Processing DeleteTablet for tablet 53352630a3c54327b176b5c595826738 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:47:10 UTC) from {username='kudu', principal='kudu/127.3.243.126@KRBTEST.COM'} at 127.0.0.1:32794
I20260619 18:47:10.587899 6216 tablet_service.cc:1514] Processing DeleteTablet for tablet 53352630a3c54327b176b5c595826738 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:47:10 UTC) from {username='kudu', principal='kudu/127.3.243.126@KRBTEST.COM'} at 127.0.0.1:39980
I20260619 18:47:10.588405 6447 tablet_replica.cc:330] T 53352630a3c54327b176b5c595826738 P b146718b0a3244aca8d5d949e5d8f13e: stopping tablet replica
I20260619 18:47:10.588593 6447 raft_consensus.cc:2229] T 53352630a3c54327b176b5c595826738 P b146718b0a3244aca8d5d949e5d8f13e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:47:10.588810 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 5872
I20260619 18:47:10.588852 6447 raft_consensus.cc:2258] T 53352630a3c54327b176b5c595826738 P b146718b0a3244aca8d5d949e5d8f13e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:47:10.589844 6447 ts_tablet_manager.cc:1905] T 53352630a3c54327b176b5c595826738 P b146718b0a3244aca8d5d949e5d8f13e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:47:10.597345 6447 ts_tablet_manager.cc:1918] T 53352630a3c54327b176b5c595826738 P b146718b0a3244aca8d5d949e5d8f13e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 18:47:10.597600 6447 log.cc:1196] T 53352630a3c54327b176b5c595826738 P b146718b0a3244aca8d5d949e5d8f13e: Deleting WAL directory at /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/minicluster-data/ts-0/wal/wals/53352630a3c54327b176b5c595826738
I20260619 18:47:10.598166 6447 ts_tablet_manager.cc:1939] T 53352630a3c54327b176b5c595826738 P b146718b0a3244aca8d5d949e5d8f13e: Deleting consensus metadata
I20260619 18:47:10.599376 5788 catalog_manager.cc:4841] TS b146718b0a3244aca8d5d949e5d8f13e (127.3.243.65:45537): tablet 53352630a3c54327b176b5c595826738 (table test-table [id=f7b4036f385d446084d100ab75a61e53]) successfully deleted
I20260619 18:47:10.600353 6449 tablet_replica.cc:330] T 53352630a3c54327b176b5c595826738 P efb278fc23954c9d86d8c54f68b6462e: stopping tablet replica
I20260619 18:47:10.600594 6449 raft_consensus.cc:2229] T 53352630a3c54327b176b5c595826738 P efb278fc23954c9d86d8c54f68b6462e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:47:10.600847 6449 raft_consensus.cc:2258] T 53352630a3c54327b176b5c595826738 P efb278fc23954c9d86d8c54f68b6462e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:47:10.601946 6449 ts_tablet_manager.cc:1905] T 53352630a3c54327b176b5c595826738 P efb278fc23954c9d86d8c54f68b6462e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:47:10.607718 6449 ts_tablet_manager.cc:1918] T 53352630a3c54327b176b5c595826738 P efb278fc23954c9d86d8c54f68b6462e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 18:47:10.607831 6449 log.cc:1196] T 53352630a3c54327b176b5c595826738 P efb278fc23954c9d86d8c54f68b6462e: Deleting WAL directory at /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/minicluster-data/ts-2/wal/wals/53352630a3c54327b176b5c595826738
I20260619 18:47:10.608242 6449 ts_tablet_manager.cc:1939] T 53352630a3c54327b176b5c595826738 P efb278fc23954c9d86d8c54f68b6462e: Deleting consensus metadata
I20260619 18:47:10.608898 6448 tablet_replica.cc:330] T 53352630a3c54327b176b5c595826738 P e3dbb92deec94038b08d065aa47a1bf1: stopping tablet replica
I20260619 18:47:10.609185 6448 raft_consensus.cc:2229] T 53352630a3c54327b176b5c595826738 P e3dbb92deec94038b08d065aa47a1bf1 [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:47:10.609287 5787 catalog_manager.cc:4841] TS efb278fc23954c9d86d8c54f68b6462e (127.3.243.67:35465): tablet 53352630a3c54327b176b5c595826738 (table test-table [id=f7b4036f385d446084d100ab75a61e53]) successfully deleted
I20260619 18:47:10.609870 6448 raft_consensus.cc:2258] T 53352630a3c54327b176b5c595826738 P e3dbb92deec94038b08d065aa47a1bf1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:47:10.610800 6448 ts_tablet_manager.cc:1905] T 53352630a3c54327b176b5c595826738 P e3dbb92deec94038b08d065aa47a1bf1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:47:10.616823 6448 ts_tablet_manager.cc:1918] T 53352630a3c54327b176b5c595826738 P e3dbb92deec94038b08d065aa47a1bf1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 18:47:10.616967 6448 log.cc:1196] T 53352630a3c54327b176b5c595826738 P e3dbb92deec94038b08d065aa47a1bf1: Deleting WAL directory at /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894810715282-4045-0/minicluster-data/ts-1/wal/wals/53352630a3c54327b176b5c595826738
Jun 19 18:47:10 dist-test-slave-m9wg krb5kdc[5756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894828, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.243.65@KRBTEST.COM
I20260619 18:47:10.617300 6448 ts_tablet_manager.cc:1939] T 53352630a3c54327b176b5c595826738 P e3dbb92deec94038b08d065aa47a1bf1: Deleting consensus metadata
I20260619 18:47:10.618183 5787 catalog_manager.cc:4841] TS e3dbb92deec94038b08d065aa47a1bf1 (127.3.243.66:35763): tablet 53352630a3c54327b176b5c595826738 (table test-table [id=f7b4036f385d446084d100ab75a61e53]) successfully deleted
I20260619 18:47:10.623112 6450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:10.590372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33472 (local address 127.3.243.65:45537)
0619 18:47:10.605929 (+ 15557us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:10.605935 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:10.608364 (+ 2429us) server_negotiation.cc:407] Connection header received
0619 18:47:10.608577 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:10.608582 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:10.608675 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:10.608857 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:10.610571 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:10.611403 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:10.613078 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:10.613498 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:10.613590 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:10.618357 (+ 4767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:10.618372 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:10.618388 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:10.618422 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:10.620945 (+ 2523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:10.621527 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:10.621534 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:10.621536 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:10.621665 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:10.621938 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:10.621941 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:10.621943 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:10.622270 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:10.622489 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:10.622656 (+ 167us) server_negotiation.cc:299] Negotiation successful
0619 18:47:10.622806 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15416,"thread_start_us":15570,"threads_started":1}
W20260619 18:47:10.623732 6000 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33472
W20260619 18:47:10.624332 4045 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 5872: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 18:47:10.624533 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 5872
I20260619 18:47:10.640609 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 6012
Jun 19 18:47:10 dist-test-slave-m9wg krb5kdc[5756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894828, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.243.66@KRBTEST.COM
I20260619 18:47:10.654202 6452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:10.641270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55680 (local address 127.3.243.66:35763)
0619 18:47:10.641743 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:10.641748 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:10.641763 (+ 15us) server_negotiation.cc:407] Connection header received
0619 18:47:10.641818 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:10.641821 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:10.641881 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:10.641973 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:10.643093 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:10.643806 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:10.644968 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:10.645351 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:10.645452 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:10.649285 (+ 3833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:10.649300 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:10.649315 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:10.649352 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:10.652052 (+ 2700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:10.652645 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:10.652652 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:10.652654 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:10.652784 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:10.653042 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:10.653045 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:10.653047 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:10.653379 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:10.653608 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:10.653782 (+ 174us) server_negotiation.cc:299] Negotiation successful
0619 18:47:10.653933 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":237,"threads_started":1}
W20260619 18:47:10.654901 6142 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55680
W20260619 18:47:10.655498 4045 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 6012: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 18:47:10.655661 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 6012
I20260619 18:47:10.673775 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 6151
Jun 19 18:47:10 dist-test-slave-m9wg krb5kdc[5756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894828, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.243.67@KRBTEST.COM
I20260619 18:47:10.688627 6453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:10.674548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34980 (local address 127.3.243.67:35465)
0619 18:47:10.674981 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:10.674985 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:10.675000 (+ 15us) server_negotiation.cc:407] Connection header received
0619 18:47:10.675049 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:10.675052 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:10.675116 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:10.675199 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:10.676518 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:10.677258 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:10.678497 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:10.679147 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:10.679268 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:10.683237 (+ 3969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:10.683251 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:10.683266 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:10.683299 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:10.686063 (+ 2764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:10.686688 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:10.686696 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:10.686698 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:10.686828 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:10.687170 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:10.687174 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:10.687176 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:10.687516 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:10.687814 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:10.688072 (+ 258us) server_negotiation.cc:299] Negotiation successful
0619 18:47:10.688312 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":206,"threads_started":1}
W20260619 18:47:10.689263 6281 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34980
W20260619 18:47:10.689817 4045 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 6151: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 18:47:10.689960 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 6151
I20260619 18:47:10.710057 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 5769
I20260619 18:47:10.711221 5841 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:47:10.828472 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 5769
2026-06-19T18:47:10Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8270 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894810715282-4045-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:47:10 dist-test-slave-m9wg krb5kdc[6457](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:47:10 dist-test-slave-m9wg krb5kdc[6457](info): set up 2 sockets
Jun 19 18:47:10 dist-test-slave-m9wg krb5kdc[6457](info): commencing operation
krb5kdc: starting...
W20260619 18:47:11.832258 4045 mini_kdc.cc:121] Time spent starting KDC: real 0.965s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894810715282-4045-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894810715282-4045-0/krb5kdc/test-user.keytab.
Jun 19 18:47:11 dist-test-slave-m9wg krb5kdc[6457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894831, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:47:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:47:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.243.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab.
Entry for principal kudu/127.3.243.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab.
I20260619 18:47:11.997185 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894810715282-4045-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894810715282-4045-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894810715282-4045-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.126:43237
--webserver_interface=127.3.243.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab
--principal=kudu/127.3.243.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:46417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.126:43237
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:47:12.308076 6470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:47:12.308406 6470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:47:12.308496 6470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:47:12.315745 6470 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:47:12.315853 6470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:47:12.315907 6470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:47:12.315964 6470 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:47:12.316007 6470 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:47:12.325843 6470 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:46417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894810715282-4045-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894810715282-4045-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.243.126:43237
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab
--principal=kudu/127.3.243.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.126:43237
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.243.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894810715282-4045-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:47:12.327132 6470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:47:12.328433 6470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:47:12.339664 6476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:12.339730 6475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:12.339701 6478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:47:12.340099 6470 server_base.cc:1034] running on GCE node
I20260619 18:47:12.341094 6470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:47:12.342480 6470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:47:12.343686 6470 hybrid_clock.cc:648] HybridClock initialized: now 1781894832343629 us; error 48 us; skew 500 ppm
I20260619 18:47:12.344025 6470 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:47:12 dist-test-slave-m9wg krb5kdc[6457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894832, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:47:12.346998 6470 init.cc:377] Logged in from keytab as kudu/127.3.243.126@KRBTEST.COM (short username kudu)
I20260619 18:47:12.348428 6470 webserver.cc:460] Webserver started at http://127.3.243.126:40893/ using document root <none> and password file <none>
I20260619 18:47:12.349097 6470 fs_manager.cc:359] Metadata directory not provided
I20260619 18:47:12.349193 6470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:47:12.349447 6470 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:47:12.352495 6470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894810715282-4045-0/minicluster-data/master-0/data/instance:
uuid: "cc332abaa4d643fba01db86cc442cdd7"
format_stamp: "Formatted at 2026-06-19 18:47:12 on dist-test-slave-m9wg"
server_key: "6401dcb3770401224ee3df1fc9043da1"
server_key_iv: "e553570aa91766e6606b1be7debe3ab7"
server_key_version: "encryptionkey@0"
I20260619 18:47:12.353217 6470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894810715282-4045-0/minicluster-data/master-0/wal/instance:
uuid: "cc332abaa4d643fba01db86cc442cdd7"
format_stamp: "Formatted at 2026-06-19 18:47:12 on dist-test-slave-m9wg"
server_key: "6401dcb3770401224ee3df1fc9043da1"
server_key_iv: "e553570aa91766e6606b1be7debe3ab7"
server_key_version: "encryptionkey@0"
I20260619 18:47:12.358124 6470 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260619 18:47:12.361469 6485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:12.362766 6470 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 18:47:12.362912 6470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894810715282-4045-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894810715282-4045-0/minicluster-data/master-0/wal
uuid: "cc332abaa4d643fba01db86cc442cdd7"
format_stamp: "Formatted at 2026-06-19 18:47:12 on dist-test-slave-m9wg"
server_key: "6401dcb3770401224ee3df1fc9043da1"
server_key_iv: "e553570aa91766e6606b1be7debe3ab7"
server_key_version: "encryptionkey@0"
I20260619 18:47:12.363075 6470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894810715282-4045-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894810715282-4045-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894810715282-4045-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:47:12.376901 6470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:47:12.380990 6470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:47:12.381217 6470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:47:12.397832 6470 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.126:43237
I20260619 18:47:12.397852 6536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.126:43237 every 8 connection(s)
I20260619 18:47:12.399219 6470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
I20260619 18:47:12.401710 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 6470
I20260619 18:47:12.401947 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894810715282-4045-0/minicluster-data/master-0/wal/instance
I20260619 18:47:12.402755 4045 external_mini_cluster.cc:1428] Setting key 4e2bf6995d2e2b0864c9f535e32e178b
I20260619 18:47:12.403623 6537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 18:47:12 dist-test-slave-m9wg krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894831, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.126@KRBTEST.COM
I20260619 18:47:12.415921 6537 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc332abaa4d643fba01db86cc442cdd7: Bootstrap starting.
I20260619 18:47:12.419287 6537 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cc332abaa4d643fba01db86cc442cdd7: Neither blocks nor log segments found. Creating new log.
I20260619 18:47:12.420378 6537 log.cc:824] T 00000000000000000000000000000000 P cc332abaa4d643fba01db86cc442cdd7: Log is configured to *not* fsync() on all Append() calls
I20260619 18:47:12.423137 6537 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc332abaa4d643fba01db86cc442cdd7: No bootstrap required, opened a new log
I20260619 18:47:12.423702 6540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:12.404839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56444 (local address 127.3.243.126:43237)
0619 18:47:12.405574 (+ 735us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:12.405586 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:12.405626 (+ 40us) server_negotiation.cc:407] Connection header received
0619 18:47:12.407145 (+ 1519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:12.407167 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:12.407566 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:12.407948 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:12.409395 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:12.410691 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:12.411879 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:12.412606 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:12.412786 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:12.416877 (+ 4091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:12.416885 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:12.416901 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:12.416944 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:12.419873 (+ 2929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:12.420579 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:12.420586 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:12.420592 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:12.420746 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:12.421096 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:12.421100 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:12.421102 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:12.421638 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:12.421915 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:12.422349 (+ 434us) server_negotiation.cc:299] Negotiation successful
0619 18:47:12.422740 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":315,"threads_started":1}
I20260619 18:47:12.430915 6537 raft_consensus.cc:348] T 00000000000000000000000000000000 P cc332abaa4d643fba01db86cc442cdd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc332abaa4d643fba01db86cc442cdd7" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 43237 } }
I20260619 18:47:12.431263 6537 raft_consensus.cc:374] T 00000000000000000000000000000000 P cc332abaa4d643fba01db86cc442cdd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:47:12.431348 6537 raft_consensus.cc:729] T 00000000000000000000000000000000 P cc332abaa4d643fba01db86cc442cdd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc332abaa4d643fba01db86cc442cdd7, State: Initialized, Role: FOLLOWER
I20260619 18:47:12.431864 6537 consensus_queue.cc:260] T 00000000000000000000000000000000 P cc332abaa4d643fba01db86cc442cdd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc332abaa4d643fba01db86cc442cdd7" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 43237 } }
I20260619 18:47:12.432082 6537 raft_consensus.cc:388] T 00000000000000000000000000000000 P cc332abaa4d643fba01db86cc442cdd7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:47:12.432215 6537 raft_consensus.cc:482] T 00000000000000000000000000000000 P cc332abaa4d643fba01db86cc442cdd7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:47:12.432363 6537 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cc332abaa4d643fba01db86cc442cdd7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:47:12.434687 6537 raft_consensus.cc:504] T 00000000000000000000000000000000 P cc332abaa4d643fba01db86cc442cdd7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc332abaa4d643fba01db86cc442cdd7" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 43237 } }
I20260619 18:47:12.435135 6537 leader_election.cc:302] T 00000000000000000000000000000000 P cc332abaa4d643fba01db86cc442cdd7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cc332abaa4d643fba01db86cc442cdd7; no voters:
I20260619 18:47:12.435676 6537 leader_election.cc:288] T 00000000000000000000000000000000 P cc332abaa4d643fba01db86cc442cdd7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:47:12.435781 6542 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cc332abaa4d643fba01db86cc442cdd7 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:47:12.436401 6542 raft_consensus.cc:686] T 00000000000000000000000000000000 P cc332abaa4d643fba01db86cc442cdd7 [term 1 LEADER]: Becoming Leader. State: Replica: cc332abaa4d643fba01db86cc442cdd7, State: Running, Role: LEADER
I20260619 18:47:12.436883 6542 consensus_queue.cc:237] T 00000000000000000000000000000000 P cc332abaa4d643fba01db86cc442cdd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc332abaa4d643fba01db86cc442cdd7" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 43237 } }
I20260619 18:47:12.437549 6537 sys_catalog.cc:564] T 00000000000000000000000000000000 P cc332abaa4d643fba01db86cc442cdd7 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:47:12.439191 6543 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc332abaa4d643fba01db86cc442cdd7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cc332abaa4d643fba01db86cc442cdd7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc332abaa4d643fba01db86cc442cdd7" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 43237 } } }
I20260619 18:47:12.439363 6544 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc332abaa4d643fba01db86cc442cdd7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cc332abaa4d643fba01db86cc442cdd7. Latest consensus state: current_term: 1 leader_uuid: "cc332abaa4d643fba01db86cc442cdd7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc332abaa4d643fba01db86cc442cdd7" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 43237 } } }
I20260619 18:47:12.439625 6543 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc332abaa4d643fba01db86cc442cdd7 [sys.catalog]: This master's current role is: LEADER
I20260619 18:47:12.439735 6544 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc332abaa4d643fba01db86cc442cdd7 [sys.catalog]: This master's current role is: LEADER
I20260619 18:47:12.440626 6549 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:47:12.446794 6549 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:47:12.455490 6549 catalog_manager.cc:1347] Generated new cluster ID: d844221218c44b7f9facbdf50acb367b
I20260619 18:47:12.455585 6549 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:47:12.468659 6549 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:47:12.470309 6549 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:47:12.481971 6549 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cc332abaa4d643fba01db86cc442cdd7: Generated new TSK 0
I20260619 18:47:12.482951 6549 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.243.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.65.keytab.
Entry for principal kudu/127.3.243.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.65.keytab.
I20260619 18:47:12.527370 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894810715282-4045-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894810715282-4045-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894810715282-4045-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894810715282-4045-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.65:0
--local_ip_for_outbound_sockets=127.3.243.65
--webserver_interface=127.3.243.65
--webserver_port=0
--tserver_master_addrs=127.3.243.126:43237
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.65.keytab
--principal=kudu/127.3.243.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:46417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:47:12.840894 6563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:47:12.841224 6563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:47:12.841327 6563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:47:12.848711 6563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:47:12.848826 6563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:47:12.848945 6563 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.65
I20260619 18:47:12.858726 6563 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:46417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894810715282-4045-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894810715282-4045-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.65.keytab
--principal=kudu/127.3.243.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894810715282-4045-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.243.65
--webserver_port=0
--tserver_master_addrs=127.3.243.126:43237
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.65
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894810715282-4045-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:47:12.859938 6563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:47:12.861276 6563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:47:12.873646 6569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:12.873670 6571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:12.874078 6568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:47:12.874374 6563 server_base.cc:1034] running on GCE node
I20260619 18:47:12.875088 6563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:47:12.876040 6563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:47:12.877311 6563 hybrid_clock.cc:648] HybridClock initialized: now 1781894832877227 us; error 52 us; skew 500 ppm
I20260619 18:47:12.877610 6563 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:47:12 dist-test-slave-m9wg krb5kdc[6457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894832, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:47:12.880609 6563 init.cc:377] Logged in from keytab as kudu/127.3.243.65@KRBTEST.COM (short username kudu)
I20260619 18:47:12.882092 6563 webserver.cc:460] Webserver started at http://127.3.243.65:41249/ using document root <none> and password file <none>
I20260619 18:47:12.882762 6563 fs_manager.cc:359] Metadata directory not provided
I20260619 18:47:12.882880 6563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:47:12.883167 6563 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:47:12.886057 6563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894810715282-4045-0/minicluster-data/ts-0/data/instance:
uuid: "97babebde9684eb3a71a4db73a978e6d"
format_stamp: "Formatted at 2026-06-19 18:47:12 on dist-test-slave-m9wg"
server_key: "0fec35527816bb3d3c7aa209dd940446"
server_key_iv: "684bb580f4911021d5511319a3882d83"
server_key_version: "encryptionkey@0"
I20260619 18:47:12.886842 6563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894810715282-4045-0/minicluster-data/ts-0/wal/instance:
uuid: "97babebde9684eb3a71a4db73a978e6d"
format_stamp: "Formatted at 2026-06-19 18:47:12 on dist-test-slave-m9wg"
server_key: "0fec35527816bb3d3c7aa209dd940446"
server_key_iv: "684bb580f4911021d5511319a3882d83"
server_key_version: "encryptionkey@0"
I20260619 18:47:12.891799 6563 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 18:47:12.895108 6578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:12.896577 6563 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:47:12.896735 6563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894810715282-4045-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894810715282-4045-0/minicluster-data/ts-0/wal
uuid: "97babebde9684eb3a71a4db73a978e6d"
format_stamp: "Formatted at 2026-06-19 18:47:12 on dist-test-slave-m9wg"
server_key: "0fec35527816bb3d3c7aa209dd940446"
server_key_iv: "684bb580f4911021d5511319a3882d83"
server_key_version: "encryptionkey@0"
I20260619 18:47:12.896916 6563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894810715282-4045-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894810715282-4045-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894810715282-4045-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:47:12.927379 6563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:47:12.931635 6563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:47:12.931921 6563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:47:12.933071 6563 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:47:12.934657 6563 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:47:12.934746 6563 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:12.934862 6563 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:47:12.934923 6563 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:12.965226 6563 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.65:43013
I20260619 18:47:12.965262 6690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.65:43013 every 8 connection(s)
I20260619 18:47:12.966514 6563 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894810715282-4045-0/minicluster-data/ts-0/data/info.pb
I20260619 18:47:12.973369 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 6563
I20260619 18:47:12.973558 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894810715282-4045-0/minicluster-data/ts-0/wal/instance
I20260619 18:47:12.974120 4045 external_mini_cluster.cc:1428] Setting key 25c61f78523c911716508823f7be2e6c
Jun 19 18:47:12 dist-test-slave-m9wg krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894832, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.65@KRBTEST.COM for kudu/127.3.243.126@KRBTEST.COM
I20260619 18:47:12.986554 6693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:12.969018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.65:50525 (local address 127.3.243.126:43237)
0619 18:47:12.969437 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:12.969442 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:12.970345 (+ 903us) server_negotiation.cc:407] Connection header received
0619 18:47:12.972193 (+ 1848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:12.972199 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:12.972284 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:12.972387 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:12.973897 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:12.974692 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:12.976558 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:12.976960 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:12.977067 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:12.980535 (+ 3468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:12.980541 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:12.980544 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:12.980577 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:12.983528 (+ 2951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:12.984263 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:12.984269 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:12.984273 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:12.984450 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:12.984851 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:12.984855 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:12.984858 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:12.985204 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:12.985333 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:12.985975 (+ 642us) server_negotiation.cc:299] Negotiation successful
0619 18:47:12.986191 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":237,"threads_started":1}
I20260619 18:47:12.987335 6694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:12.969504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:43237 (local address 127.3.243.65:50525)
0619 18:47:12.970156 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:12.970202 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:12.971900 (+ 1698us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:12.972636 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:12.972644 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:12.973128 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:47:12.973739 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:12.973758 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:12.974846 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:12.974850 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:12.976428 (+ 1578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:12.976440 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:12.977167 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:12.977174 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:12.977405 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:12.978009 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:47:12.978032 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:47:12.980305 (+ 2273us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:47:12.983683 (+ 3378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:12.983690 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:12.983704 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:12.984133 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:12.984555 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:12.984558 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:12.984560 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:12.984745 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:12.985311 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:47:12.985317 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:47:12.985638 (+ 321us) client_negotiation.cc:769] Sending connection context
0619 18:47:12.985859 (+ 221us) client_negotiation.cc:240] Negotiation successful
0619 18:47:12.986246 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":261,"threads_started":1}
I20260619 18:47:12.988827 6691 heartbeater.cc:347] Connected to a master server at 127.3.243.126:43237
I20260619 18:47:12.989112 6691 heartbeater.cc:464] Registering TS with master...
I20260619 18:47:12.989811 6691 heartbeater.cc:511] Master 127.3.243.126:43237 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.3.243.66@KRBTEST.COM; defaulting to no policy
I20260619 18:47:12.991814 6502 ts_manager.cc:194] Registered new tserver with Master: 97babebde9684eb3a71a4db73a978e6d (127.3.243.65:43013)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.66@KRBTEST.COM" created.
I20260619 18:47:12.993371 6502 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.65@KRBTEST.COM'} at 127.3.243.65:50525
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.66.keytab.
Entry for principal kudu/127.3.243.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.66.keytab.
I20260619 18:47:13.013103 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894810715282-4045-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894810715282-4045-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894810715282-4045-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894810715282-4045-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.66:0
--local_ip_for_outbound_sockets=127.3.243.66
--webserver_interface=127.3.243.66
--webserver_port=0
--tserver_master_addrs=127.3.243.126:43237
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.66.keytab
--principal=kudu/127.3.243.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:46417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:47:13.324584 6697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:47:13.324889 6697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:47:13.325016 6697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:47:13.332335 6697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:47:13.332461 6697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:47:13.332615 6697 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.66
I20260619 18:47:13.342629 6697 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:46417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894810715282-4045-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894810715282-4045-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.66.keytab
--principal=kudu/127.3.243.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894810715282-4045-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.243.66
--webserver_port=0
--tserver_master_addrs=127.3.243.126:43237
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.66
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894810715282-4045-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:47:13.343935 6697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:47:13.345249 6697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:47:13.357728 6705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:13.357688 6703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:13.357998 6702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:47:13.358976 6697 server_base.cc:1034] running on GCE node
I20260619 18:47:13.359575 6697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:47:13.360610 6697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:47:13.361824 6697 hybrid_clock.cc:648] HybridClock initialized: now 1781894833361763 us; error 49 us; skew 500 ppm
I20260619 18:47:13.362131 6697 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:47:13 dist-test-slave-m9wg krb5kdc[6457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894833, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:47:13.365109 6697 init.cc:377] Logged in from keytab as kudu/127.3.243.66@KRBTEST.COM (short username kudu)
I20260619 18:47:13.366547 6697 webserver.cc:460] Webserver started at http://127.3.243.66:45091/ using document root <none> and password file <none>
I20260619 18:47:13.367225 6697 fs_manager.cc:359] Metadata directory not provided
I20260619 18:47:13.367388 6697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:47:13.367676 6697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:47:13.370549 6697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894810715282-4045-0/minicluster-data/ts-1/data/instance:
uuid: "13f5f0a4c337455b9b2355bdb2b085e3"
format_stamp: "Formatted at 2026-06-19 18:47:13 on dist-test-slave-m9wg"
server_key: "31604899f78afc9e5244e38b45a074e5"
server_key_iv: "e8e94ac2141f1fc1456e8b5449f95e9b"
server_key_version: "encryptionkey@0"
I20260619 18:47:13.371296 6697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894810715282-4045-0/minicluster-data/ts-1/wal/instance:
uuid: "13f5f0a4c337455b9b2355bdb2b085e3"
format_stamp: "Formatted at 2026-06-19 18:47:13 on dist-test-slave-m9wg"
server_key: "31604899f78afc9e5244e38b45a074e5"
server_key_iv: "e8e94ac2141f1fc1456e8b5449f95e9b"
server_key_version: "encryptionkey@0"
I20260619 18:47:13.376399 6697 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260619 18:47:13.379706 6712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:13.381009 6697 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:47:13.381160 6697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894810715282-4045-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894810715282-4045-0/minicluster-data/ts-1/wal
uuid: "13f5f0a4c337455b9b2355bdb2b085e3"
format_stamp: "Formatted at 2026-06-19 18:47:13 on dist-test-slave-m9wg"
server_key: "31604899f78afc9e5244e38b45a074e5"
server_key_iv: "e8e94ac2141f1fc1456e8b5449f95e9b"
server_key_version: "encryptionkey@0"
I20260619 18:47:13.381359 6697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894810715282-4045-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894810715282-4045-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894810715282-4045-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:47:13.396123 6697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:47:13.400394 6697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:47:13.400657 6697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:47:13.409358 6697 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:47:13.410738 6697 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:47:13.410813 6697 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:13.410916 6697 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:47:13.410967 6697 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:13.439989 6697 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.66:45755
I20260619 18:47:13.440057 6824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.66:45755 every 8 connection(s)
I20260619 18:47:13.441262 6697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894810715282-4045-0/minicluster-data/ts-1/data/info.pb
I20260619 18:47:13.449931 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 6697
I20260619 18:47:13.450109 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894810715282-4045-0/minicluster-data/ts-1/wal/instance
I20260619 18:47:13.450500 4045 external_mini_cluster.cc:1428] Setting key 1b4a62b3dda0d6b4786ec9a16f8a5ecf
Jun 19 18:47:13 dist-test-slave-m9wg krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894833, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.66@KRBTEST.COM for kudu/127.3.243.126@KRBTEST.COM
I20260619 18:47:13.461109 6693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:13.443687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.66:41219 (local address 127.3.243.126:43237)
0619 18:47:13.443898 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:13.443904 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:13.445140 (+ 1236us) server_negotiation.cc:407] Connection header received
0619 18:47:13.446964 (+ 1824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:13.446969 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:13.447041 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:13.447129 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:13.448592 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:13.449376 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:13.450949 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:13.451386 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:13.451488 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:13.455191 (+ 3703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:13.455197 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:13.455201 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:13.455236 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:13.458217 (+ 2981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:13.458903 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:13.458907 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:13.458909 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:13.459043 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:13.459450 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:13.459453 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:13.459455 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:13.459713 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:13.459841 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:13.460556 (+ 715us) server_negotiation.cc:299] Negotiation successful
0619 18:47:13.460780 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 18:47:13.462085 6827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:13.444250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:43237 (local address 127.3.243.66:41219)
0619 18:47:13.444954 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:13.444991 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:13.446696 (+ 1705us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:13.447320 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:13.447328 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:13.447819 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:47:13.448443 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:13.448461 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:13.449519 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:13.449523 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:13.450797 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:13.450809 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:13.451604 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:13.451611 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:13.451896 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:13.452598 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:47:13.452620 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:47:13.454973 (+ 2353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:47:13.458346 (+ 3373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:13.458355 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:13.458374 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:13.458781 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:13.459143 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:13.459148 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:13.459152 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:13.459338 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:13.459830 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:47:13.459838 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:47:13.460202 (+ 364us) client_negotiation.cc:769] Sending connection context
0619 18:47:13.460448 (+ 246us) client_negotiation.cc:240] Negotiation successful
0619 18:47:13.460970 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":264,"threads_started":1}
I20260619 18:47:13.463477 6825 heartbeater.cc:347] Connected to a master server at 127.3.243.126:43237
I20260619 18:47:13.463829 6825 heartbeater.cc:464] Registering TS with master...
I20260619 18:47:13.464800 6825 heartbeater.cc:511] Master 127.3.243.126:43237 requested a full tablet report, sending...
I20260619 18:47:13.466360 6502 ts_manager.cc:194] Registered new tserver with Master: 13f5f0a4c337455b9b2355bdb2b085e3 (127.3.243.66:45755)
WARNING: no policy specified for kudu/127.3.243.67@KRBTEST.COM; defaulting to no policy
I20260619 18:47:13.467242 6502 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.66@KRBTEST.COM'} at 127.3.243.66:41219
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.67.keytab.
Entry for principal kudu/127.3.243.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.67.keytab.
I20260619 18:47:13.489121 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894810715282-4045-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894810715282-4045-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894810715282-4045-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894810715282-4045-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.67:0
--local_ip_for_outbound_sockets=127.3.243.67
--webserver_interface=127.3.243.67
--webserver_port=0
--tserver_master_addrs=127.3.243.126:43237
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.67.keytab
--principal=kudu/127.3.243.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:46417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:47:13.799050 6830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:47:13.799322 6830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:47:13.799422 6830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:47:13.806763 6830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:47:13.806875 6830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:47:13.806989 6830 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.67
I20260619 18:47:13.816728 6830 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:46417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894810715282-4045-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894810715282-4045-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.67.keytab
--principal=kudu/127.3.243.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894810715282-4045-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.243.67
--webserver_port=0
--tserver_master_addrs=127.3.243.126:43237
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.67
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894810715282-4045-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:47:13.817978 6830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:47:13.819181 6830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:47:13.831231 6836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:13.831303 6835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:13.831322 6838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:47:13.831831 6830 server_base.cc:1034] running on GCE node
I20260619 18:47:13.832448 6830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:47:13.833293 6830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:47:13.834473 6830 hybrid_clock.cc:648] HybridClock initialized: now 1781894833834421 us; error 45 us; skew 500 ppm
I20260619 18:47:13.834753 6830 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:47:13 dist-test-slave-m9wg krb5kdc[6457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894833, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:47:13.837706 6830 init.cc:377] Logged in from keytab as kudu/127.3.243.67@KRBTEST.COM (short username kudu)
I20260619 18:47:13.838990 6830 webserver.cc:460] Webserver started at http://127.3.243.67:39607/ using document root <none> and password file <none>
I20260619 18:47:13.839610 6830 fs_manager.cc:359] Metadata directory not provided
I20260619 18:47:13.839701 6830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:47:13.839941 6830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:47:13.842645 6830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894810715282-4045-0/minicluster-data/ts-2/data/instance:
uuid: "e75ae4ab705e49f4ab7fe72d0c362cfc"
format_stamp: "Formatted at 2026-06-19 18:47:13 on dist-test-slave-m9wg"
server_key: "1966104f81cbbf19bf8452dc8295cc74"
server_key_iv: "6f480b0748dd01771be95519b95d2c6e"
server_key_version: "encryptionkey@0"
I20260619 18:47:13.843381 6830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894810715282-4045-0/minicluster-data/ts-2/wal/instance:
uuid: "e75ae4ab705e49f4ab7fe72d0c362cfc"
format_stamp: "Formatted at 2026-06-19 18:47:13 on dist-test-slave-m9wg"
server_key: "1966104f81cbbf19bf8452dc8295cc74"
server_key_iv: "6f480b0748dd01771be95519b95d2c6e"
server_key_version: "encryptionkey@0"
I20260619 18:47:13.848250 6830 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260619 18:47:13.851519 6845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:13.852802 6830 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 18:47:13.852941 6830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894810715282-4045-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894810715282-4045-0/minicluster-data/ts-2/wal
uuid: "e75ae4ab705e49f4ab7fe72d0c362cfc"
format_stamp: "Formatted at 2026-06-19 18:47:13 on dist-test-slave-m9wg"
server_key: "1966104f81cbbf19bf8452dc8295cc74"
server_key_iv: "6f480b0748dd01771be95519b95d2c6e"
server_key_version: "encryptionkey@0"
I20260619 18:47:13.853119 6830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894810715282-4045-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894810715282-4045-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894810715282-4045-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:47:13.871001 6830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:47:13.874702 6830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:47:13.874943 6830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:47:13.875779 6830 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:47:13.877108 6830 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:47:13.877180 6830 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:13.877300 6830 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:47:13.877351 6830 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:13.905737 6830 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.67:42943
I20260619 18:47:13.905776 6957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.67:42943 every 8 connection(s)
I20260619 18:47:13.906935 6830 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894810715282-4045-0/minicluster-data/ts-2/data/info.pb
I20260619 18:47:13.915230 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 6830
I20260619 18:47:13.915448 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894810715282-4045-0/minicluster-data/ts-2/wal/instance
I20260619 18:47:13.916033 4045 external_mini_cluster.cc:1428] Setting key 334c3a65abe1953395ae78f6a8bfe65e
Jun 19 18:47:13 dist-test-slave-m9wg krb5kdc[6457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894833, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.67@KRBTEST.COM for kudu/127.3.243.126@KRBTEST.COM
I20260619 18:47:13.925189 6693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:13.909334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.67:58733 (local address 127.3.243.126:43237)
0619 18:47:13.909502 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:13.909507 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:13.910673 (+ 1166us) server_negotiation.cc:407] Connection header received
0619 18:47:13.912465 (+ 1792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:13.912469 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:13.912539 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:13.912624 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:13.914035 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:13.914763 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:13.916893 (+ 2130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:13.917339 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:13.917473 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:13.920670 (+ 3197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:13.920674 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:13.920678 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:13.920708 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:13.922850 (+ 2142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:13.923432 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:13.923435 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:13.923437 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:13.923542 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:13.923843 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:13.923846 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:13.923848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:13.924086 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:13.924228 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:13.924792 (+ 564us) server_negotiation.cc:299] Negotiation successful
0619 18:47:13.924955 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 18:47:13.926069 6960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:13.909828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:43237 (local address 127.3.243.67:58733)
0619 18:47:13.910510 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:13.910551 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:13.912217 (+ 1666us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:13.912822 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:13.912830 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:13.913341 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:47:13.913905 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:13.913922 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:13.914913 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:13.914918 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:13.916750 (+ 1832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:13.916762 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:13.917542 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:13.917547 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:13.917752 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:13.918391 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:47:13.918418 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:47:13.920497 (+ 2079us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:47:13.922960 (+ 2463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:13.922967 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:13.922978 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:13.923346 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:13.923609 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:13.923612 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:13.923614 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:13.923770 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:13.924199 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:47:13.924204 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:47:13.924477 (+ 273us) client_negotiation.cc:769] Sending connection context
0619 18:47:13.924686 (+ 209us) client_negotiation.cc:240] Negotiation successful
0619 18:47:13.925032 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":258,"threads_started":1}
I20260619 18:47:13.927227 6958 heartbeater.cc:347] Connected to a master server at 127.3.243.126:43237
I20260619 18:47:13.927502 6958 heartbeater.cc:464] Registering TS with master...
I20260619 18:47:13.928186 6958 heartbeater.cc:511] Master 127.3.243.126:43237 requested a full tablet report, sending...
I20260619 18:47:13.929517 6502 ts_manager.cc:194] Registered new tserver with Master: e75ae4ab705e49f4ab7fe72d0c362cfc (127.3.243.67:42943)
I20260619 18:47:13.930464 6502 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.67@KRBTEST.COM'} at 127.3.243.67:58733
I20260619 18:47:13.931140 4045 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 18:47:13.957937 6693 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:47:13.955842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56454 (local address 127.3.243.126:43237)
0619 18:47:13.956014 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:13.956020 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:13.956366 (+ 346us) server_negotiation.cc:407] Connection header received
0619 18:47:13.956506 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:13.956510 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:13.956581 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:13.956668 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:13.957700 (+ 1032us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56454: BlockingRecv error: recv got EOF from 127.0.0.1:56454 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 18:47:13.961122 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 6563
W20260619 18:47:13.963626 4045 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 6563: Not authorized: Client connection negotiation failed: client connection to 127.3.243.65:43013: server requires authentication, but client does not have Kerberos credentials available
I20260619 18:47:13.963812 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 6563
I20260619 18:47:13.976251 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 6697
W20260619 18:47:13.978487 4045 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 6697: Not authorized: Client connection negotiation failed: client connection to 127.3.243.66:45755: server requires authentication, but client does not have Kerberos credentials available
I20260619 18:47:13.978642 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 6697
I20260619 18:47:13.990377 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 6830
W20260619 18:47:13.992940 4045 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 6830: Not authorized: Client connection negotiation failed: client connection to 127.3.243.67:42943: server requires authentication, but client does not have Kerberos credentials available
I20260619 18:47:13.993047 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 6830
I20260619 18:47:14.005050 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 6470
I20260619 18:47:14.006080 6532 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:47:14.106383 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 6470
2026-06-19T18:47:14Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3275 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781894810715282-4045-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:47:14 dist-test-slave-m9wg krb5kdc[6975](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:47:14 dist-test-slave-m9wg krb5kdc[6975](info): set up 2 sockets
Jun 19 18:47:14 dist-test-slave-m9wg krb5kdc[6975](info): commencing operation
krb5kdc: starting...
W20260619 18:47:15.092196 4045 mini_kdc.cc:121] Time spent starting KDC: real 0.949s user 0.001s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781894810715282-4045-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781894810715282-4045-0/krb5kdc/test-user.keytab.
Jun 19 18:47:15 dist-test-slave-m9wg krb5kdc[6975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894835, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:47:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:47:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.243.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab.
Entry for principal kudu/127.3.243.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab.
I20260619 18:47:15.253118 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781894810715282-4045-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781894810715282-4045-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781894810715282-4045-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.126:34155
--webserver_interface=127.3.243.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab
--principal=kudu/127.3.243.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:41951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.126:34155
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:47:15.553746 6988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:47:15.554018 6988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:47:15.554144 6988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:47:15.561167 6988 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:47:15.561265 6988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:47:15.561338 6988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:47:15.561415 6988 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:47:15.561477 6988 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:47:15.570886 6988 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:41951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781894810715282-4045-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781894810715282-4045-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.243.126:34155
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab
--principal=kudu/127.3.243.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.126:34155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.243.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781894810715282-4045-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:47:15.572170 6988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:47:15.573426 6988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:47:15.584301 6993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:15.584360 6994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:15.584533 6996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:47:15.585095 6988 server_base.cc:1034] running on GCE node
I20260619 18:47:15.585811 6988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:47:15.587031 6988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:47:15.588239 6988 hybrid_clock.cc:648] HybridClock initialized: now 1781894835588200 us; error 26 us; skew 500 ppm
I20260619 18:47:15.588585 6988 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:47:15 dist-test-slave-m9wg krb5kdc[6975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894835, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:47:15.591442 6988 init.cc:377] Logged in from keytab as kudu/127.3.243.126@KRBTEST.COM (short username kudu)
I20260619 18:47:15.592800 6988 webserver.cc:460] Webserver started at http://127.3.243.126:41429/ using document root <none> and password file <none>
I20260619 18:47:15.593397 6988 fs_manager.cc:359] Metadata directory not provided
I20260619 18:47:15.593492 6988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:47:15.593729 6988 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:47:15.596508 6988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781894810715282-4045-0/minicluster-data/master-0/data/instance:
uuid: "065aded6e1c14b4b91fd00990dbe4a10"
format_stamp: "Formatted at 2026-06-19 18:47:15 on dist-test-slave-m9wg"
server_key: "c364178b86e5540c880d926fb496be1d"
server_key_iv: "6e62b7e150baeac1746acb2c3f466463"
server_key_version: "encryptionkey@0"
I20260619 18:47:15.597193 6988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781894810715282-4045-0/minicluster-data/master-0/wal/instance:
uuid: "065aded6e1c14b4b91fd00990dbe4a10"
format_stamp: "Formatted at 2026-06-19 18:47:15 on dist-test-slave-m9wg"
server_key: "c364178b86e5540c880d926fb496be1d"
server_key_iv: "6e62b7e150baeac1746acb2c3f466463"
server_key_version: "encryptionkey@0"
I20260619 18:47:15.601884 6988 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 18:47:15.604936 7003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:15.606091 6988 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 18:47:15.606240 6988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781894810715282-4045-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781894810715282-4045-0/minicluster-data/master-0/wal
uuid: "065aded6e1c14b4b91fd00990dbe4a10"
format_stamp: "Formatted at 2026-06-19 18:47:15 on dist-test-slave-m9wg"
server_key: "c364178b86e5540c880d926fb496be1d"
server_key_iv: "6e62b7e150baeac1746acb2c3f466463"
server_key_version: "encryptionkey@0"
I20260619 18:47:15.606410 6988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781894810715282-4045-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781894810715282-4045-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781894810715282-4045-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:47:15.655102 6988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:47:15.659327 6988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:47:15.659590 6988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:47:15.675901 6988 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.126:34155
I20260619 18:47:15.675942 7054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.126:34155 every 8 connection(s)
I20260619 18:47:15.677239 6988 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
I20260619 18:47:15.680603 7055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:47:15.683844 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 6988
I20260619 18:47:15.684015 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781894810715282-4045-0/minicluster-data/master-0/wal/instance
I20260619 18:47:15.684633 4045 external_mini_cluster.cc:1428] Setting key e94e3da1accf7e26a227b8459ebc9437
I20260619 18:47:15.692411 7055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 065aded6e1c14b4b91fd00990dbe4a10: Bootstrap starting.
I20260619 18:47:15.695865 7055 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 065aded6e1c14b4b91fd00990dbe4a10: Neither blocks nor log segments found. Creating new log.
I20260619 18:47:15.697113 7055 log.cc:824] T 00000000000000000000000000000000 P 065aded6e1c14b4b91fd00990dbe4a10: Log is configured to *not* fsync() on all Append() calls
Jun 19 18:47:15 dist-test-slave-m9wg krb5kdc[6975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894835, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.126@KRBTEST.COM
I20260619 18:47:15.699733 7055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 065aded6e1c14b4b91fd00990dbe4a10: No bootstrap required, opened a new log
I20260619 18:47:15.706986 7058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:15.686330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41488 (local address 127.3.243.126:34155)
0619 18:47:15.687077 (+ 747us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:15.687089 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:15.687128 (+ 39us) server_negotiation.cc:407] Connection header received
0619 18:47:15.688670 (+ 1542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:15.688691 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:15.689100 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:15.689417 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:15.691096 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:15.692380 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:15.694056 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:15.694688 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:15.694826 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:15.699239 (+ 4413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:15.699247 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:15.699260 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:15.699292 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:15.702834 (+ 3542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:15.703429 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:15.703440 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:15.703449 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:15.703653 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:15.703949 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:15.703953 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:15.703960 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:15.704638 (+ 678us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:15.704924 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:15.705232 (+ 308us) server_negotiation.cc:299] Negotiation successful
0619 18:47:15.705670 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":323,"threads_started":1}
I20260619 18:47:15.709318 7055 raft_consensus.cc:348] T 00000000000000000000000000000000 P 065aded6e1c14b4b91fd00990dbe4a10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "065aded6e1c14b4b91fd00990dbe4a10" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 34155 } }
I20260619 18:47:15.709719 7055 raft_consensus.cc:374] T 00000000000000000000000000000000 P 065aded6e1c14b4b91fd00990dbe4a10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:47:15.709812 7055 raft_consensus.cc:729] T 00000000000000000000000000000000 P 065aded6e1c14b4b91fd00990dbe4a10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 065aded6e1c14b4b91fd00990dbe4a10, State: Initialized, Role: FOLLOWER
I20260619 18:47:15.710381 7055 consensus_queue.cc:260] T 00000000000000000000000000000000 P 065aded6e1c14b4b91fd00990dbe4a10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "065aded6e1c14b4b91fd00990dbe4a10" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 34155 } }
I20260619 18:47:15.710585 7055 raft_consensus.cc:388] T 00000000000000000000000000000000 P 065aded6e1c14b4b91fd00990dbe4a10 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:47:15.710649 4045 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:47:15.710708 7055 raft_consensus.cc:482] T 00000000000000000000000000000000 P 065aded6e1c14b4b91fd00990dbe4a10 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:47:15.710861 7055 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 065aded6e1c14b4b91fd00990dbe4a10 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:47:15.712891 7055 raft_consensus.cc:504] T 00000000000000000000000000000000 P 065aded6e1c14b4b91fd00990dbe4a10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "065aded6e1c14b4b91fd00990dbe4a10" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 34155 } }
I20260619 18:47:15.713320 7055 leader_election.cc:302] T 00000000000000000000000000000000 P 065aded6e1c14b4b91fd00990dbe4a10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 065aded6e1c14b4b91fd00990dbe4a10; no voters:
I20260619 18:47:15.713863 7055 leader_election.cc:288] T 00000000000000000000000000000000 P 065aded6e1c14b4b91fd00990dbe4a10 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:47:15.713984 7060 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 065aded6e1c14b4b91fd00990dbe4a10 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:47:15.715624 7055 sys_catalog.cc:564] T 00000000000000000000000000000000 P 065aded6e1c14b4b91fd00990dbe4a10 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:47:15.717963 7060 raft_consensus.cc:686] T 00000000000000000000000000000000 P 065aded6e1c14b4b91fd00990dbe4a10 [term 1 LEADER]: Becoming Leader. State: Replica: 065aded6e1c14b4b91fd00990dbe4a10, State: Running, Role: LEADER
I20260619 18:47:15.718613 7060 consensus_queue.cc:237] T 00000000000000000000000000000000 P 065aded6e1c14b4b91fd00990dbe4a10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "065aded6e1c14b4b91fd00990dbe4a10" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 34155 } }
I20260619 18:47:15.723238 7062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 065aded6e1c14b4b91fd00990dbe4a10 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 065aded6e1c14b4b91fd00990dbe4a10. Latest consensus state: current_term: 1 leader_uuid: "065aded6e1c14b4b91fd00990dbe4a10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "065aded6e1c14b4b91fd00990dbe4a10" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 34155 } } }
I20260619 18:47:15.723557 7062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 065aded6e1c14b4b91fd00990dbe4a10 [sys.catalog]: This master's current role is: LEADER
I20260619 18:47:15.724045 7061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 065aded6e1c14b4b91fd00990dbe4a10 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "065aded6e1c14b4b91fd00990dbe4a10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "065aded6e1c14b4b91fd00990dbe4a10" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 34155 } } }
I20260619 18:47:15.724360 7061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 065aded6e1c14b4b91fd00990dbe4a10 [sys.catalog]: This master's current role is: LEADER
I20260619 18:47:15.734346 7072 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:47:15.739094 7072 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:47:15.750439 7072 catalog_manager.cc:1347] Generated new cluster ID: 9f34416590b54433a77e4b49ecdf2a33
I20260619 18:47:15.750558 7072 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:47:15.755095 7058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:15.747552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41500 (local address 127.3.243.126:34155)
0619 18:47:15.747725 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:15.747732 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:15.748261 (+ 529us) server_negotiation.cc:407] Connection header received
0619 18:47:15.748539 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:15.748545 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:15.748626 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:15.748755 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:15.749983 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:15.751022 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:15.752248 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:15.752656 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:15.752762 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:15.753753 (+ 991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:15.753757 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:15.753760 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:47:15.754347 (+ 587us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:15.754408 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:47:15.754435 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:15.754523 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:15.754674 (+ 151us) server_negotiation.cc:299] Negotiation successful
0619 18:47:15.754765 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 18:47:15.771377 7072 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:47:15.772733 7072 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:47:15.781100 7072 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 065aded6e1c14b4b91fd00990dbe4a10: Generated new TSK 0
I20260619 18:47:15.781883 7072 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:47:15.791080 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 6988
I20260619 18:47:15.792316 7050 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:47:15.881419 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 6988
2026-06-19T18:47:15Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1770 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894810715282-4045-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:47:15 dist-test-slave-m9wg krb5kdc[7090](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:47:15 dist-test-slave-m9wg krb5kdc[7090](info): set up 2 sockets
Jun 19 18:47:15 dist-test-slave-m9wg krb5kdc[7090](info): commencing operation
krb5kdc: starting...
W20260619 18:47:16.864363 4045 mini_kdc.cc:121] Time spent starting KDC: real 0.951s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894810715282-4045-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894810715282-4045-0/krb5kdc/test-user.keytab.
Jun 19 18:47:16 dist-test-slave-m9wg krb5kdc[7090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:47:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:47:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.243.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab.
Entry for principal kudu/127.3.243.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab.
I20260619 18:47:17.036878 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894810715282-4045-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894810715282-4045-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894810715282-4045-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.126:34199
--webserver_interface=127.3.243.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab
--principal=kudu/127.3.243.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:33459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.126:34199
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:47:17.343801 7103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:47:17.344082 7103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:47:17.344215 7103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:47:17.351423 7103 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:47:17.351541 7103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:47:17.351586 7103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:47:17.351641 7103 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:47:17.351684 7103 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:47:17.361560 7103 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:33459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894810715282-4045-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894810715282-4045-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.243.126:34199
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab
--principal=kudu/127.3.243.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.126:34199
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.243.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894810715282-4045-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:47:17.362871 7103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:47:17.364108 7103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:47:17.375319 7108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:17.375263 7109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:17.375233 7111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:47:17.376030 7103 server_base.cc:1034] running on GCE node
I20260619 18:47:17.376791 7103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:47:17.378079 7103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:47:17.379267 7103 hybrid_clock.cc:648] HybridClock initialized: now 1781894837379230 us; error 30 us; skew 500 ppm
I20260619 18:47:17.379550 7103 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:47:17 dist-test-slave-m9wg krb5kdc[7090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894837, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:47:17.386987 7103 init.cc:377] Logged in from keytab as kudu/127.3.243.126@KRBTEST.COM (short username kudu)
I20260619 18:47:17.388374 7103 webserver.cc:460] Webserver started at http://127.3.243.126:33233/ using document root <none> and password file <none>
I20260619 18:47:17.389026 7103 fs_manager.cc:359] Metadata directory not provided
I20260619 18:47:17.389137 7103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:47:17.389417 7103 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:47:17.392320 7103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894810715282-4045-0/minicluster-data/master-0/data/instance:
uuid: "13d8b0f3474c48e599617b921683520d"
format_stamp: "Formatted at 2026-06-19 18:47:17 on dist-test-slave-m9wg"
server_key: "e44c8e40bdb8144e2cd68448db01c9cd"
server_key_iv: "479911fa2c5d728e0321442e92d8e21b"
server_key_version: "encryptionkey@0"
I20260619 18:47:17.393153 7103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894810715282-4045-0/minicluster-data/master-0/wal/instance:
uuid: "13d8b0f3474c48e599617b921683520d"
format_stamp: "Formatted at 2026-06-19 18:47:17 on dist-test-slave-m9wg"
server_key: "e44c8e40bdb8144e2cd68448db01c9cd"
server_key_iv: "479911fa2c5d728e0321442e92d8e21b"
server_key_version: "encryptionkey@0"
I20260619 18:47:17.398267 7103 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 18:47:17.401610 7118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:17.402977 7103 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 18:47:17.403132 7103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894810715282-4045-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894810715282-4045-0/minicluster-data/master-0/wal
uuid: "13d8b0f3474c48e599617b921683520d"
format_stamp: "Formatted at 2026-06-19 18:47:17 on dist-test-slave-m9wg"
server_key: "e44c8e40bdb8144e2cd68448db01c9cd"
server_key_iv: "479911fa2c5d728e0321442e92d8e21b"
server_key_version: "encryptionkey@0"
I20260619 18:47:17.403318 7103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894810715282-4045-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894810715282-4045-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894810715282-4045-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:47:17.424726 7103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:47:17.428588 7103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:47:17.428838 7103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:47:17.445308 7103 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.126:34199
I20260619 18:47:17.445348 7169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.126:34199 every 8 connection(s)
I20260619 18:47:17.446583 7103 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
I20260619 18:47:17.450001 7170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:47:17.452139 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 7103
I20260619 18:47:17.452360 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894810715282-4045-0/minicluster-data/master-0/wal/instance
I20260619 18:47:17.452960 4045 external_mini_cluster.cc:1428] Setting key ce66a46a97923e6406fcae62f12be3e7
I20260619 18:47:17.460950 7170 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13d8b0f3474c48e599617b921683520d: Bootstrap starting.
I20260619 18:47:17.464387 7170 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 13d8b0f3474c48e599617b921683520d: Neither blocks nor log segments found. Creating new log.
I20260619 18:47:17.465602 7170 log.cc:824] T 00000000000000000000000000000000 P 13d8b0f3474c48e599617b921683520d: Log is configured to *not* fsync() on all Append() calls
Jun 19 18:47:17 dist-test-slave-m9wg krb5kdc[7090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.126@KRBTEST.COM
I20260619 18:47:17.468286 7170 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13d8b0f3474c48e599617b921683520d: No bootstrap required, opened a new log
I20260619 18:47:17.474968 7173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:17.454860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46112 (local address 127.3.243.126:34199)
0619 18:47:17.455590 (+ 730us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:17.455600 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:17.455640 (+ 40us) server_negotiation.cc:407] Connection header received
0619 18:47:17.457241 (+ 1601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:17.457263 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:17.457644 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:17.457970 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:17.459569 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:17.460909 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:17.462556 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:17.463214 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:17.463332 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:17.467798 (+ 4466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:17.467805 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:17.467819 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:17.467857 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:17.470724 (+ 2867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:17.471340 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:17.471347 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:17.471352 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:17.471529 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:17.471962 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:17.471968 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:17.471972 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:17.472667 (+ 695us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:17.472970 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:17.473300 (+ 330us) server_negotiation.cc:299] Negotiation successful
0619 18:47:17.473717 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":339,"threads_started":1}
I20260619 18:47:17.476864 7170 raft_consensus.cc:348] T 00000000000000000000000000000000 P 13d8b0f3474c48e599617b921683520d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13d8b0f3474c48e599617b921683520d" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 34199 } }
I20260619 18:47:17.477180 7170 raft_consensus.cc:374] T 00000000000000000000000000000000 P 13d8b0f3474c48e599617b921683520d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:47:17.477277 7170 raft_consensus.cc:729] T 00000000000000000000000000000000 P 13d8b0f3474c48e599617b921683520d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13d8b0f3474c48e599617b921683520d, State: Initialized, Role: FOLLOWER
I20260619 18:47:17.477877 7170 consensus_queue.cc:260] T 00000000000000000000000000000000 P 13d8b0f3474c48e599617b921683520d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13d8b0f3474c48e599617b921683520d" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 34199 } }
I20260619 18:47:17.478086 7170 raft_consensus.cc:388] T 00000000000000000000000000000000 P 13d8b0f3474c48e599617b921683520d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:47:17.478202 7170 raft_consensus.cc:482] T 00000000000000000000000000000000 P 13d8b0f3474c48e599617b921683520d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:47:17.478358 7170 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 13d8b0f3474c48e599617b921683520d [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:47:17.480350 7170 raft_consensus.cc:504] T 00000000000000000000000000000000 P 13d8b0f3474c48e599617b921683520d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13d8b0f3474c48e599617b921683520d" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 34199 } }
I20260619 18:47:17.480773 7170 leader_election.cc:302] T 00000000000000000000000000000000 P 13d8b0f3474c48e599617b921683520d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 13d8b0f3474c48e599617b921683520d; no voters:
I20260619 18:47:17.481285 7170 leader_election.cc:288] T 00000000000000000000000000000000 P 13d8b0f3474c48e599617b921683520d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:47:17.481401 7175 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 13d8b0f3474c48e599617b921683520d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:47:17.481997 7175 raft_consensus.cc:686] T 00000000000000000000000000000000 P 13d8b0f3474c48e599617b921683520d [term 1 LEADER]: Becoming Leader. State: Replica: 13d8b0f3474c48e599617b921683520d, State: Running, Role: LEADER
I20260619 18:47:17.482556 7175 consensus_queue.cc:237] T 00000000000000000000000000000000 P 13d8b0f3474c48e599617b921683520d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13d8b0f3474c48e599617b921683520d" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 34199 } }
I20260619 18:47:17.483196 7170 sys_catalog.cc:564] T 00000000000000000000000000000000 P 13d8b0f3474c48e599617b921683520d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:47:17.485241 7176 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13d8b0f3474c48e599617b921683520d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "13d8b0f3474c48e599617b921683520d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13d8b0f3474c48e599617b921683520d" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 34199 } } }
I20260619 18:47:17.485296 7177 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13d8b0f3474c48e599617b921683520d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 13d8b0f3474c48e599617b921683520d. Latest consensus state: current_term: 1 leader_uuid: "13d8b0f3474c48e599617b921683520d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13d8b0f3474c48e599617b921683520d" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 34199 } } }
I20260619 18:47:17.485579 7176 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13d8b0f3474c48e599617b921683520d [sys.catalog]: This master's current role is: LEADER
I20260619 18:47:17.485639 7177 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13d8b0f3474c48e599617b921683520d [sys.catalog]: This master's current role is: LEADER
I20260619 18:47:17.488240 7182 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:47:17.493384 7182 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:47:17.501726 7182 catalog_manager.cc:1347] Generated new cluster ID: fd8874990ce14130bfc349ea1a030d25
I20260619 18:47:17.501813 7182 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:47:17.509095 7182 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:47:17.510303 7182 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:47:17.518290 7182 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 13d8b0f3474c48e599617b921683520d: Generated new TSK 0
I20260619 18:47:17.519049 7182 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.243.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.65.keytab.
Entry for principal kudu/127.3.243.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.65.keytab.
I20260619 18:47:17.567710 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894810715282-4045-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894810715282-4045-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894810715282-4045-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894810715282-4045-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.65:0
--local_ip_for_outbound_sockets=127.3.243.65
--webserver_interface=127.3.243.65
--webserver_port=0
--tserver_master_addrs=127.3.243.126:34199
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.65.keytab
--principal=kudu/127.3.243.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:33459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:47:17.876696 7196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:47:17.876950 7196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:47:17.877067 7196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:47:17.884107 7196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:47:17.884240 7196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:47:17.884402 7196 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.65
I20260619 18:47:17.894039 7196 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:33459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894810715282-4045-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894810715282-4045-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.65.keytab
--principal=kudu/127.3.243.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894810715282-4045-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.243.65
--webserver_port=0
--tserver_master_addrs=127.3.243.126:34199
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.65
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894810715282-4045-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:47:17.895308 7196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:47:17.896618 7196 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:47:17.908916 7201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:17.908991 7202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:17.909407 7204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:47:17.909789 7196 server_base.cc:1034] running on GCE node
I20260619 18:47:17.910351 7196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:47:17.911224 7196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:47:17.912422 7196 hybrid_clock.cc:648] HybridClock initialized: now 1781894837912364 us; error 42 us; skew 500 ppm
I20260619 18:47:17.912719 7196 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:47:17 dist-test-slave-m9wg krb5kdc[7090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894837, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:47:17.915719 7196 init.cc:377] Logged in from keytab as kudu/127.3.243.65@KRBTEST.COM (short username kudu)
I20260619 18:47:17.917237 7196 webserver.cc:460] Webserver started at http://127.3.243.65:44839/ using document root <none> and password file <none>
I20260619 18:47:17.917912 7196 fs_manager.cc:359] Metadata directory not provided
I20260619 18:47:17.918028 7196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:47:17.918313 7196 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:47:17.921257 7196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894810715282-4045-0/minicluster-data/ts-0/data/instance:
uuid: "bd022c886a0e4070a1428a7f3eb01527"
format_stamp: "Formatted at 2026-06-19 18:47:17 on dist-test-slave-m9wg"
server_key: "550cc6140eeeb17de4ca7f16bc5a2202"
server_key_iv: "fde6ce4bdb118466488c30db5bca7211"
server_key_version: "encryptionkey@0"
I20260619 18:47:17.922013 7196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894810715282-4045-0/minicluster-data/ts-0/wal/instance:
uuid: "bd022c886a0e4070a1428a7f3eb01527"
format_stamp: "Formatted at 2026-06-19 18:47:17 on dist-test-slave-m9wg"
server_key: "550cc6140eeeb17de4ca7f16bc5a2202"
server_key_iv: "fde6ce4bdb118466488c30db5bca7211"
server_key_version: "encryptionkey@0"
I20260619 18:47:17.927282 7196 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 18:47:17.930666 7211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:17.931973 7196 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 18:47:17.932143 7196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894810715282-4045-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894810715282-4045-0/minicluster-data/ts-0/wal
uuid: "bd022c886a0e4070a1428a7f3eb01527"
format_stamp: "Formatted at 2026-06-19 18:47:17 on dist-test-slave-m9wg"
server_key: "550cc6140eeeb17de4ca7f16bc5a2202"
server_key_iv: "fde6ce4bdb118466488c30db5bca7211"
server_key_version: "encryptionkey@0"
I20260619 18:47:17.932407 7196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894810715282-4045-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894810715282-4045-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894810715282-4045-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:47:17.962046 7196 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:47:17.965826 7196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:47:17.966068 7196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:47:17.966933 7196 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:47:17.968245 7196 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:47:17.968323 7196 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:17.968439 7196 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:47:17.968498 7196 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:17.997385 7196 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.65:40091
I20260619 18:47:17.997433 7323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.65:40091 every 8 connection(s)
I20260619 18:47:17.998603 7196 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894810715282-4045-0/minicluster-data/ts-0/data/info.pb
I20260619 18:47:18.006078 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 7196
I20260619 18:47:18.006265 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894810715282-4045-0/minicluster-data/ts-0/wal/instance
I20260619 18:47:18.006982 4045 external_mini_cluster.cc:1428] Setting key 7f26ec3e24c49b57cee0553c96700828
Jun 19 18:47:18 dist-test-slave-m9wg krb5kdc[7090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894837, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.65@KRBTEST.COM for kudu/127.3.243.126@KRBTEST.COM
I20260619 18:47:18.019296 7326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:18.001002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.65:33151 (local address 127.3.243.126:34199)
0619 18:47:18.001431 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:18.001438 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:18.002401 (+ 963us) server_negotiation.cc:407] Connection header received
0619 18:47:18.004186 (+ 1785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:18.004193 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:18.004273 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:18.004373 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:18.005896 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:18.006710 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:18.008583 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:18.009124 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:18.009222 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:18.013280 (+ 4058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:18.013286 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:18.013290 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:18.013330 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:18.016076 (+ 2746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:18.016949 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:18.016954 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:18.016956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:18.017071 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:18.017500 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:18.017503 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:18.017505 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:18.017775 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:18.017947 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:18.018693 (+ 746us) server_negotiation.cc:299] Negotiation successful
0619 18:47:18.018937 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":199,"threads_started":1}
I20260619 18:47:18.020429 7327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:18.001588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:34199 (local address 127.3.243.65:33151)
0619 18:47:18.002249 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:18.002288 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:18.003886 (+ 1598us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:18.004595 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:18.004603 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:18.005121 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:47:18.005730 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:18.005756 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:18.006874 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:18.006880 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:18.008451 (+ 1571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:18.008464 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:18.009357 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:18.009365 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:18.009661 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:18.010345 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:47:18.010376 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:47:18.013043 (+ 2667us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:47:18.016243 (+ 3200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:18.016253 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:18.016272 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:18.016844 (+ 572us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:18.017157 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:18.017161 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:18.017164 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:18.017405 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:18.017908 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:47:18.017917 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:47:18.018287 (+ 370us) client_negotiation.cc:769] Sending connection context
0619 18:47:18.018602 (+ 315us) client_negotiation.cc:240] Negotiation successful
0619 18:47:18.019151 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":265,"threads_started":1}
I20260619 18:47:18.022156 7324 heartbeater.cc:347] Connected to a master server at 127.3.243.126:34199
I20260619 18:47:18.022473 7324 heartbeater.cc:464] Registering TS with master...
I20260619 18:47:18.023296 7324 heartbeater.cc:511] Master 127.3.243.126:34199 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.3.243.66@KRBTEST.COM; defaulting to no policy
I20260619 18:47:18.025744 7135 ts_manager.cc:194] Registered new tserver with Master: bd022c886a0e4070a1428a7f3eb01527 (127.3.243.65:40091)
I20260619 18:47:18.027263 7135 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.65@KRBTEST.COM'} at 127.3.243.65:33151
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.66.keytab.
Entry for principal kudu/127.3.243.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.66.keytab.
I20260619 18:47:18.049189 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894810715282-4045-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894810715282-4045-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894810715282-4045-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894810715282-4045-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.66:0
--local_ip_for_outbound_sockets=127.3.243.66
--webserver_interface=127.3.243.66
--webserver_port=0
--tserver_master_addrs=127.3.243.126:34199
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.66.keytab
--principal=kudu/127.3.243.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:33459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:47:18.357801 7330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:47:18.358088 7330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:47:18.358191 7330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:47:18.365341 7330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:47:18.365448 7330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:47:18.365561 7330 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.66
I20260619 18:47:18.375389 7330 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:33459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894810715282-4045-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894810715282-4045-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.66.keytab
--principal=kudu/127.3.243.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894810715282-4045-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.243.66
--webserver_port=0
--tserver_master_addrs=127.3.243.126:34199
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.66
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894810715282-4045-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:47:18.376708 7330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:47:18.378007 7330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:47:18.390364 7335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:18.390857 7338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:18.391041 7336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:47:18.391577 7330 server_base.cc:1034] running on GCE node
I20260619 18:47:18.392200 7330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:47:18.393159 7330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:47:18.394363 7330 hybrid_clock.cc:648] HybridClock initialized: now 1781894838394301 us; error 55 us; skew 500 ppm
I20260619 18:47:18.394667 7330 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:47:18 dist-test-slave-m9wg krb5kdc[7090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894838, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:47:18.397671 7330 init.cc:377] Logged in from keytab as kudu/127.3.243.66@KRBTEST.COM (short username kudu)
I20260619 18:47:18.399112 7330 webserver.cc:460] Webserver started at http://127.3.243.66:44061/ using document root <none> and password file <none>
I20260619 18:47:18.399758 7330 fs_manager.cc:359] Metadata directory not provided
I20260619 18:47:18.399880 7330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:47:18.400197 7330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:47:18.403021 7330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894810715282-4045-0/minicluster-data/ts-1/data/instance:
uuid: "41458238453047dbab926f8dba3804a5"
format_stamp: "Formatted at 2026-06-19 18:47:18 on dist-test-slave-m9wg"
server_key: "fb57ac0218ba9f74ab1dc4c3839980e0"
server_key_iv: "ddb9de4506c2de395aab6c42e8cbd068"
server_key_version: "encryptionkey@0"
I20260619 18:47:18.403771 7330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894810715282-4045-0/minicluster-data/ts-1/wal/instance:
uuid: "41458238453047dbab926f8dba3804a5"
format_stamp: "Formatted at 2026-06-19 18:47:18 on dist-test-slave-m9wg"
server_key: "fb57ac0218ba9f74ab1dc4c3839980e0"
server_key_iv: "ddb9de4506c2de395aab6c42e8cbd068"
server_key_version: "encryptionkey@0"
I20260619 18:47:18.408979 7330 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260619 18:47:18.412389 7345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:18.413695 7330 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 18:47:18.413852 7330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894810715282-4045-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894810715282-4045-0/minicluster-data/ts-1/wal
uuid: "41458238453047dbab926f8dba3804a5"
format_stamp: "Formatted at 2026-06-19 18:47:18 on dist-test-slave-m9wg"
server_key: "fb57ac0218ba9f74ab1dc4c3839980e0"
server_key_iv: "ddb9de4506c2de395aab6c42e8cbd068"
server_key_version: "encryptionkey@0"
I20260619 18:47:18.414027 7330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894810715282-4045-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894810715282-4045-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894810715282-4045-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:47:18.446972 7330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:47:18.451237 7330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:47:18.451509 7330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:47:18.452504 7330 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:47:18.453830 7330 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:47:18.453910 7330 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:18.454041 7330 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:47:18.454093 7330 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:18.483847 7330 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.66:36475
I20260619 18:47:18.483896 7457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.66:36475 every 8 connection(s)
I20260619 18:47:18.485250 7330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894810715282-4045-0/minicluster-data/ts-1/data/info.pb
I20260619 18:47:18.486394 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 7330
I20260619 18:47:18.486564 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894810715282-4045-0/minicluster-data/ts-1/wal/instance
I20260619 18:47:18.487087 4045 external_mini_cluster.cc:1428] Setting key d17d86283290b55e8137eee9a9b3aaca
Jun 19 18:47:18 dist-test-slave-m9wg krb5kdc[7090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894838, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.66@KRBTEST.COM for kudu/127.3.243.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.243.67@KRBTEST.COM; defaulting to no policy
I20260619 18:47:18.506852 7326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:18.488041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.66:36327 (local address 127.3.243.126:34199)
0619 18:47:18.488333 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:18.488339 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:18.489694 (+ 1355us) server_negotiation.cc:407] Connection header received
0619 18:47:18.492205 (+ 2511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:18.492210 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:18.492294 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:18.492386 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:18.494336 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:18.495087 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:18.496724 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:18.497359 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:18.497490 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:18.500769 (+ 3279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:18.500776 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:18.500780 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:18.500817 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:18.503890 (+ 3073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:18.504694 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:18.504699 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:18.504703 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:18.504851 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:18.505239 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:18.505245 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:18.505248 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:18.505604 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:18.505751 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:18.506409 (+ 658us) server_negotiation.cc:299] Negotiation successful
0619 18:47:18.506582 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260619 18:47:18.507865 7460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:18.488700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:34199 (local address 127.3.243.66:36327)
0619 18:47:18.489515 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:18.489573 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:18.491844 (+ 2271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:18.492655 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:18.492668 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:18.493369 (+ 701us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:47:18.494163 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:18.494188 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:18.495241 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:18.495246 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:18.496579 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:18.496592 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:18.497555 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:18.497560 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:18.497750 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:18.498395 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:47:18.498422 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:47:18.500547 (+ 2125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:47:18.504039 (+ 3492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:18.504049 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:18.504069 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:18.504565 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:18.504950 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:18.504953 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:18.504955 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:18.505121 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:18.505711 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:47:18.505717 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:47:18.506053 (+ 336us) client_negotiation.cc:769] Sending connection context
0619 18:47:18.506292 (+ 239us) client_negotiation.cc:240] Negotiation successful
0619 18:47:18.506684 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":325,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.67@KRBTEST.COM" created.
I20260619 18:47:18.509397 7458 heartbeater.cc:347] Connected to a master server at 127.3.243.126:34199
I20260619 18:47:18.509673 7458 heartbeater.cc:464] Registering TS with master...
I20260619 18:47:18.510392 7458 heartbeater.cc:511] Master 127.3.243.126:34199 requested a full tablet report, sending...
I20260619 18:47:18.511834 7135 ts_manager.cc:194] Registered new tserver with Master: 41458238453047dbab926f8dba3804a5 (127.3.243.66:36475)
I20260619 18:47:18.512826 7135 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.66@KRBTEST.COM'} at 127.3.243.66:36327
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.67.keytab.
Entry for principal kudu/127.3.243.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.67.keytab.
I20260619 18:47:18.529112 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894810715282-4045-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894810715282-4045-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894810715282-4045-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894810715282-4045-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.67:0
--local_ip_for_outbound_sockets=127.3.243.67
--webserver_interface=127.3.243.67
--webserver_port=0
--tserver_master_addrs=127.3.243.126:34199
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.67.keytab
--principal=kudu/127.3.243.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:33459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:47:18.843513 7463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:47:18.843782 7463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:47:18.843925 7463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:47:18.851044 7463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:47:18.851169 7463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:47:18.851322 7463 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.67
I20260619 18:47:18.860913 7463 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:33459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894810715282-4045-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894810715282-4045-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.67.keytab
--principal=kudu/127.3.243.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894810715282-4045-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.243.67
--webserver_port=0
--tserver_master_addrs=127.3.243.126:34199
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.67
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894810715282-4045-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:47:18.862129 7463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:47:18.863372 7463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:47:18.875590 7469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:18.875780 7471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:18.875566 7468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:47:18.876400 7463 server_base.cc:1034] running on GCE node
I20260619 18:47:18.876986 7463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:47:18.877863 7463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:47:18.879062 7463 hybrid_clock.cc:648] HybridClock initialized: now 1781894838879015 us; error 33 us; skew 500 ppm
I20260619 18:47:18.879356 7463 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:47:18 dist-test-slave-m9wg krb5kdc[7090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894838, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:47:18.882382 7463 init.cc:377] Logged in from keytab as kudu/127.3.243.67@KRBTEST.COM (short username kudu)
I20260619 18:47:18.883756 7463 webserver.cc:460] Webserver started at http://127.3.243.67:46505/ using document root <none> and password file <none>
I20260619 18:47:18.884469 7463 fs_manager.cc:359] Metadata directory not provided
I20260619 18:47:18.884588 7463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:47:18.884857 7463 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:47:18.887666 7463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894810715282-4045-0/minicluster-data/ts-2/data/instance:
uuid: "ba935dcf592448a18108b27fd5846a90"
format_stamp: "Formatted at 2026-06-19 18:47:18 on dist-test-slave-m9wg"
server_key: "715ff7c37e06ec36c2c4275ed2308d13"
server_key_iv: "ecbb945a393391f8ce5fd7ec070b253d"
server_key_version: "encryptionkey@0"
I20260619 18:47:18.888437 7463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894810715282-4045-0/minicluster-data/ts-2/wal/instance:
uuid: "ba935dcf592448a18108b27fd5846a90"
format_stamp: "Formatted at 2026-06-19 18:47:18 on dist-test-slave-m9wg"
server_key: "715ff7c37e06ec36c2c4275ed2308d13"
server_key_iv: "ecbb945a393391f8ce5fd7ec070b253d"
server_key_version: "encryptionkey@0"
I20260619 18:47:18.893326 7463 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 18:47:18.896831 7478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:18.898115 7463 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 18:47:18.898244 7463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894810715282-4045-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894810715282-4045-0/minicluster-data/ts-2/wal
uuid: "ba935dcf592448a18108b27fd5846a90"
format_stamp: "Formatted at 2026-06-19 18:47:18 on dist-test-slave-m9wg"
server_key: "715ff7c37e06ec36c2c4275ed2308d13"
server_key_iv: "ecbb945a393391f8ce5fd7ec070b253d"
server_key_version: "encryptionkey@0"
I20260619 18:47:18.898411 7463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894810715282-4045-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894810715282-4045-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894810715282-4045-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:47:18.932122 7463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:47:18.936408 7463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:47:18.936686 7463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:47:18.937625 7463 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:47:18.938993 7463 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:47:18.939066 7463 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:18.939158 7463 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:47:18.939203 7463 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:18.969184 7463 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.67:40871
I20260619 18:47:18.969229 7590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.67:40871 every 8 connection(s)
I20260619 18:47:18.970409 7463 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894810715282-4045-0/minicluster-data/ts-2/data/info.pb
I20260619 18:47:18.976266 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 7463
I20260619 18:47:18.976455 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894810715282-4045-0/minicluster-data/ts-2/wal/instance
I20260619 18:47:18.976892 4045 external_mini_cluster.cc:1428] Setting key 5b75dde9542cc61ce8ee0d74f81aa739
Jun 19 18:47:18 dist-test-slave-m9wg krb5kdc[7090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894838, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.67@KRBTEST.COM for kudu/127.3.243.126@KRBTEST.COM
I20260619 18:47:18.990340 7326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:18.973046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.67:56819 (local address 127.3.243.126:34199)
0619 18:47:18.973239 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:18.973245 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:18.974431 (+ 1186us) server_negotiation.cc:407] Connection header received
0619 18:47:18.976203 (+ 1772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:18.976209 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:18.976299 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:18.976407 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:18.978439 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:18.979468 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:18.981045 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:18.981442 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:18.981553 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:18.985069 (+ 3516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:18.985075 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:18.985079 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:18.985116 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:18.987505 (+ 2389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:18.988212 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:18.988216 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:18.988218 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:18.988353 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:18.988693 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:18.988696 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:18.988697 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:18.988998 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:18.989103 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:18.989759 (+ 656us) server_negotiation.cc:299] Negotiation successful
0619 18:47:18.989973 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 18:47:18.991351 7593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:18.973587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:34199 (local address 127.3.243.67:56819)
0619 18:47:18.974247 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:18.974290 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:18.975900 (+ 1610us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:18.976775 (+ 875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:18.976787 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:18.977495 (+ 708us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:47:18.978242 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:18.978265 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:18.979595 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:18.979601 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:18.980912 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:18.980923 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:18.981661 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:18.981668 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:18.981903 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:18.982524 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:47:18.982555 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:47:18.984845 (+ 2290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:47:18.987642 (+ 2797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:18.987648 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:18.987661 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:18.988077 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:18.988437 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:18.988441 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:18.988443 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:18.988616 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:18.989098 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:47:18.989105 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:47:18.989436 (+ 331us) client_negotiation.cc:769] Sending connection context
0619 18:47:18.989657 (+ 221us) client_negotiation.cc:240] Negotiation successful
0619 18:47:18.990080 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":251,"threads_started":1}
I20260619 18:47:18.992697 7591 heartbeater.cc:347] Connected to a master server at 127.3.243.126:34199
I20260619 18:47:18.992993 7591 heartbeater.cc:464] Registering TS with master...
I20260619 18:47:18.993722 7591 heartbeater.cc:511] Master 127.3.243.126:34199 requested a full tablet report, sending...
I20260619 18:47:18.995122 7135 ts_manager.cc:194] Registered new tserver with Master: ba935dcf592448a18108b27fd5846a90 (127.3.243.67:40871)
I20260619 18:47:18.995893 7135 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.67@KRBTEST.COM'} at 127.3.243.67:56819
I20260619 18:47:19.004542 4045 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 18:47:19 dist-test-slave-m9wg krb5kdc[7090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894839, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
I20260619 18:47:19.030903 7324 heartbeater.cc:503] Master 127.3.243.126:34199 was elected leader, sending a full tablet report...
Jun 19 18:47:19 dist-test-slave-m9wg krb5kdc[7090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894839, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.3.243.126@KRBTEST.COM
I20260619 18:47:19.048906 7326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:19.035823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46120 (local address 127.3.243.126:34199)
0619 18:47:19.035995 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:19.036000 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:19.036527 (+ 527us) server_negotiation.cc:407] Connection header received
0619 18:47:19.036716 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:19.036720 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:19.036789 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:19.036895 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:19.038169 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:19.038900 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:19.040125 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:19.040564 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:19.040676 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:19.044122 (+ 3446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:19.044127 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:19.044130 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:19.044221 (+ 91us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:19.046455 (+ 2234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:19.047039 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:19.047045 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:19.047049 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:19.047191 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:19.047536 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:19.047541 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:19.047544 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:19.047854 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:19.048078 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:19.048398 (+ 320us) server_negotiation.cc:299] Negotiation successful
0619 18:47:19.048611 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260619 18:47:19.049418 7135 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:46120
I20260619 18:47:19.050911 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 7196
Jun 19 18:47:19 dist-test-slave-m9wg krb5kdc[7090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894839, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.3.243.65@KRBTEST.COM
I20260619 18:47:19.066331 7602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:19.052067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52174 (local address 127.3.243.65:40091)
0619 18:47:19.052714 (+ 647us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:19.052721 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:19.052745 (+ 24us) server_negotiation.cc:407] Connection header received
0619 18:47:19.052797 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:19.052803 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:19.053015 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:19.053134 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:19.054166 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:19.055124 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:19.056261 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:19.056834 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:19.056926 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:19.060731 (+ 3805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:19.060755 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:19.060774 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:19.060811 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:19.063475 (+ 2664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:19.064204 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:19.064214 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:19.064218 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:19.064396 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:19.064837 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:19.064840 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:19.064843 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:19.065210 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:19.065479 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:19.065796 (+ 317us) server_negotiation.cc:299] Negotiation successful
0619 18:47:19.066011 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":342,"threads_started":1}
W20260619 18:47:19.067643 7319 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:52174
W20260619 18:47:19.068825 4045 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 7196: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 18:47:19.069041 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 7196
I20260619 18:47:19.082093 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 7330
Jun 19 18:47:19 dist-test-slave-m9wg krb5kdc[7090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894839, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.3.243.66@KRBTEST.COM
I20260619 18:47:19.097200 7603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:19.083010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35152 (local address 127.3.243.66:36475)
0619 18:47:19.083514 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:19.083522 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:19.083548 (+ 26us) server_negotiation.cc:407] Connection header received
0619 18:47:19.083608 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:19.083614 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:19.083838 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:19.083997 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:19.085205 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:19.086187 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:19.087416 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:19.087977 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:19.088095 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:19.092010 (+ 3915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:19.092022 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:19.092039 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:19.092072 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:19.094691 (+ 2619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:19.095276 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:19.095282 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:19.095285 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:19.095416 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:19.095675 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:19.095678 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:19.095680 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:19.096043 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:19.096305 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:19.096622 (+ 317us) server_negotiation.cc:299] Negotiation successful
0619 18:47:19.096822 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":226,"threads_started":1}
W20260619 18:47:19.098381 7453 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:35152
W20260619 18:47:19.099395 4045 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 7330: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 18:47:19.099537 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 7330
I20260619 18:47:19.112049 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 7463
Jun 19 18:47:19 dist-test-slave-m9wg krb5kdc[7090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894839, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.3.243.67@KRBTEST.COM
I20260619 18:47:19.127322 7604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:19.113106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50936 (local address 127.3.243.67:40871)
0619 18:47:19.113759 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:19.113765 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:19.113796 (+ 31us) server_negotiation.cc:407] Connection header received
0619 18:47:19.113857 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:19.113863 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:19.114096 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:19.114219 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:19.115462 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:19.116531 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:19.117713 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:19.118237 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:19.118340 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:19.122357 (+ 4017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:19.122369 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:19.122382 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:19.122414 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:19.125013 (+ 2599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:19.125629 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:19.125635 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:19.125638 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:19.125752 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:19.126018 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:19.126021 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:19.126023 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:19.126342 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:19.126551 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:19.126770 (+ 219us) server_negotiation.cc:299] Negotiation successful
0619 18:47:19.126960 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":340,"threads_started":1}
W20260619 18:47:19.128662 7586 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:50936
W20260619 18:47:19.129733 4045 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 7463: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 18:47:19.129909 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 7463
I20260619 18:47:19.141901 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 7103
I20260619 18:47:19.143018 7165 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:47:19.246306 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 7103
2026-06-19T18:47:19Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3369 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894810715282-4045-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:47:19 dist-test-slave-m9wg krb5kdc[7608](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:47:19 dist-test-slave-m9wg krb5kdc[7608](info): set up 2 sockets
Jun 19 18:47:19 dist-test-slave-m9wg krb5kdc[7608](info): commencing operation
krb5kdc: starting...
W20260619 18:47:20.244216 4045 mini_kdc.cc:121] Time spent starting KDC: real 0.961s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894810715282-4045-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894810715282-4045-0/krb5kdc/test-user.keytab.
Jun 19 18:47:20 dist-test-slave-m9wg krb5kdc[7608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894840, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:47:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:47:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.243.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab.
Entry for principal kudu/127.3.243.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab.
I20260619 18:47:20.419530 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894810715282-4045-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894810715282-4045-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894810715282-4045-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.126:40059
--webserver_interface=127.3.243.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab
--principal=kudu/127.3.243.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:35045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.126:40059
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:47:20.729794 7621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:47:20.730080 7621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:47:20.730228 7621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:47:20.737308 7621 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:47:20.737413 7621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:47:20.737493 7621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:47:20.737566 7621 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:47:20.737638 7621 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:47:20.747272 7621 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:35045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894810715282-4045-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894810715282-4045-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.243.126:40059
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab
--principal=kudu/127.3.243.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.126:40059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.243.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894810715282-4045-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:47:20.748538 7621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:47:20.749814 7621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:47:20.760824 7626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:20.760867 7629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:20.760797 7627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:47:20.761721 7621 server_base.cc:1034] running on GCE node
I20260619 18:47:20.762467 7621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:47:20.763873 7621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:47:20.765076 7621 hybrid_clock.cc:648] HybridClock initialized: now 1781894840765015 us; error 50 us; skew 500 ppm
I20260619 18:47:20.765420 7621 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:47:20 dist-test-slave-m9wg krb5kdc[7608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894840, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:47:20.768344 7621 init.cc:377] Logged in from keytab as kudu/127.3.243.126@KRBTEST.COM (short username kudu)
I20260619 18:47:20.769698 7621 webserver.cc:460] Webserver started at http://127.3.243.126:34335/ using document root <none> and password file <none>
I20260619 18:47:20.770352 7621 fs_manager.cc:359] Metadata directory not provided
I20260619 18:47:20.770463 7621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:47:20.770744 7621 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:47:20.773649 7621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894810715282-4045-0/minicluster-data/master-0/data/instance:
uuid: "2a32f78ad3dc42d3a2bd358cd6383d77"
format_stamp: "Formatted at 2026-06-19 18:47:20 on dist-test-slave-m9wg"
server_key: "64b079667306ed302b13ce55e72f7767"
server_key_iv: "0ae8d548dd6475601716095fff76f64a"
server_key_version: "encryptionkey@0"
I20260619 18:47:20.774396 7621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894810715282-4045-0/minicluster-data/master-0/wal/instance:
uuid: "2a32f78ad3dc42d3a2bd358cd6383d77"
format_stamp: "Formatted at 2026-06-19 18:47:20 on dist-test-slave-m9wg"
server_key: "64b079667306ed302b13ce55e72f7767"
server_key_iv: "0ae8d548dd6475601716095fff76f64a"
server_key_version: "encryptionkey@0"
I20260619 18:47:20.779332 7621 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260619 18:47:20.782640 7636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:20.783998 7621 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 18:47:20.784204 7621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894810715282-4045-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894810715282-4045-0/minicluster-data/master-0/wal
uuid: "2a32f78ad3dc42d3a2bd358cd6383d77"
format_stamp: "Formatted at 2026-06-19 18:47:20 on dist-test-slave-m9wg"
server_key: "64b079667306ed302b13ce55e72f7767"
server_key_iv: "0ae8d548dd6475601716095fff76f64a"
server_key_version: "encryptionkey@0"
I20260619 18:47:20.784399 7621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894810715282-4045-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894810715282-4045-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894810715282-4045-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:47:20.811174 7621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:47:20.814839 7621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:47:20.815093 7621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:47:20.831735 7621 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.126:40059
I20260619 18:47:20.831774 7687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.126:40059 every 8 connection(s)
I20260619 18:47:20.833046 7621 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
I20260619 18:47:20.836105 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 7621
I20260619 18:47:20.836375 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894810715282-4045-0/minicluster-data/master-0/wal/instance
I20260619 18:47:20.836969 7688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:47:20.837175 4045 external_mini_cluster.cc:1428] Setting key 4e9a534c592cc71a0139e47fcd055d4d
I20260619 18:47:20.850752 7688 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a32f78ad3dc42d3a2bd358cd6383d77: Bootstrap starting.
Jun 19 18:47:20 dist-test-slave-m9wg krb5kdc[7608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894840, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.126@KRBTEST.COM
I20260619 18:47:20.854350 7688 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a32f78ad3dc42d3a2bd358cd6383d77: Neither blocks nor log segments found. Creating new log.
I20260619 18:47:20.855630 7688 log.cc:824] T 00000000000000000000000000000000 P 2a32f78ad3dc42d3a2bd358cd6383d77: Log is configured to *not* fsync() on all Append() calls
I20260619 18:47:20.858560 7688 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a32f78ad3dc42d3a2bd358cd6383d77: No bootstrap required, opened a new log
I20260619 18:47:20.859508 7691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:20.839130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36700 (local address 127.3.243.126:40059)
0619 18:47:20.839900 (+ 770us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:20.839919 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:20.839966 (+ 47us) server_negotiation.cc:407] Connection header received
0619 18:47:20.841841 (+ 1875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:20.841867 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:20.842362 (+ 495us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:20.842784 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:20.843985 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:20.845921 (+ 1936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:20.847106 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:20.848135 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:20.848355 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:20.852046 (+ 3691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:20.852061 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:20.852079 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:20.852136 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:20.855802 (+ 3666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:20.856493 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:20.856500 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:20.856508 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:20.856714 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:20.857034 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:20.857038 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:20.857040 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:20.857528 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:20.857804 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:20.858191 (+ 387us) server_negotiation.cc:299] Negotiation successful
0619 18:47:20.858570 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":302,"threads_started":1}
I20260619 18:47:20.866312 7688 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a32f78ad3dc42d3a2bd358cd6383d77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a32f78ad3dc42d3a2bd358cd6383d77" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 40059 } }
I20260619 18:47:20.866668 7688 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a32f78ad3dc42d3a2bd358cd6383d77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:47:20.866765 7688 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a32f78ad3dc42d3a2bd358cd6383d77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a32f78ad3dc42d3a2bd358cd6383d77, State: Initialized, Role: FOLLOWER
I20260619 18:47:20.867296 7688 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a32f78ad3dc42d3a2bd358cd6383d77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a32f78ad3dc42d3a2bd358cd6383d77" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 40059 } }
I20260619 18:47:20.867513 7688 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a32f78ad3dc42d3a2bd358cd6383d77 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:47:20.867645 7688 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a32f78ad3dc42d3a2bd358cd6383d77 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:47:20.867784 7688 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a32f78ad3dc42d3a2bd358cd6383d77 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:47:20.869942 7688 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a32f78ad3dc42d3a2bd358cd6383d77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a32f78ad3dc42d3a2bd358cd6383d77" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 40059 } }
I20260619 18:47:20.870420 7688 leader_election.cc:302] T 00000000000000000000000000000000 P 2a32f78ad3dc42d3a2bd358cd6383d77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a32f78ad3dc42d3a2bd358cd6383d77; no voters:
I20260619 18:47:20.870959 7688 leader_election.cc:288] T 00000000000000000000000000000000 P 2a32f78ad3dc42d3a2bd358cd6383d77 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:47:20.871070 7693 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a32f78ad3dc42d3a2bd358cd6383d77 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:47:20.871675 7693 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a32f78ad3dc42d3a2bd358cd6383d77 [term 1 LEADER]: Becoming Leader. State: Replica: 2a32f78ad3dc42d3a2bd358cd6383d77, State: Running, Role: LEADER
I20260619 18:47:20.872270 7693 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a32f78ad3dc42d3a2bd358cd6383d77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a32f78ad3dc42d3a2bd358cd6383d77" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 40059 } }
I20260619 18:47:20.872895 7688 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a32f78ad3dc42d3a2bd358cd6383d77 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:47:20.875370 7694 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a32f78ad3dc42d3a2bd358cd6383d77 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2a32f78ad3dc42d3a2bd358cd6383d77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a32f78ad3dc42d3a2bd358cd6383d77" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 40059 } } }
I20260619 18:47:20.875401 7695 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a32f78ad3dc42d3a2bd358cd6383d77 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a32f78ad3dc42d3a2bd358cd6383d77. Latest consensus state: current_term: 1 leader_uuid: "2a32f78ad3dc42d3a2bd358cd6383d77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a32f78ad3dc42d3a2bd358cd6383d77" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 40059 } } }
I20260619 18:47:20.875818 7695 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a32f78ad3dc42d3a2bd358cd6383d77 [sys.catalog]: This master's current role is: LEADER
I20260619 18:47:20.875816 7694 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a32f78ad3dc42d3a2bd358cd6383d77 [sys.catalog]: This master's current role is: LEADER
I20260619 18:47:20.877317 7701 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:47:20.882490 7701 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:47:20.891261 7701 catalog_manager.cc:1347] Generated new cluster ID: c5f19a2da6eb48d29534ea6701912977
I20260619 18:47:20.891383 7701 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:47:20.901744 7701 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:47:20.903007 7701 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:47:20.915112 7701 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2a32f78ad3dc42d3a2bd358cd6383d77: Generated new TSK 0
I20260619 18:47:20.915913 7701 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.243.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.65.keytab.
Entry for principal kudu/127.3.243.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.65.keytab.
I20260619 18:47:20.955868 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894810715282-4045-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894810715282-4045-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894810715282-4045-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894810715282-4045-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.65:0
--local_ip_for_outbound_sockets=127.3.243.65
--webserver_interface=127.3.243.65
--webserver_port=0
--tserver_master_addrs=127.3.243.126:40059
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.65.keytab
--principal=kudu/127.3.243.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:35045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:47:21.274401 7714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:47:21.274691 7714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:47:21.274807 7714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:47:21.282123 7714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:47:21.282229 7714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:47:21.282366 7714 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.65
I20260619 18:47:21.292181 7714 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:35045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894810715282-4045-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894810715282-4045-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.65.keytab
--principal=kudu/127.3.243.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894810715282-4045-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.243.65
--webserver_port=0
--tserver_master_addrs=127.3.243.126:40059
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.65
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894810715282-4045-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:47:21.293481 7714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:47:21.294776 7714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:47:21.307046 7720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:21.307122 7722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:21.307293 7719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:47:21.307861 7714 server_base.cc:1034] running on GCE node
I20260619 18:47:21.308456 7714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:47:21.309317 7714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:47:21.310504 7714 hybrid_clock.cc:648] HybridClock initialized: now 1781894841310454 us; error 42 us; skew 500 ppm
I20260619 18:47:21.310781 7714 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:47:21 dist-test-slave-m9wg krb5kdc[7608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894841, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:47:21.313673 7714 init.cc:377] Logged in from keytab as kudu/127.3.243.65@KRBTEST.COM (short username kudu)
I20260619 18:47:21.315007 7714 webserver.cc:460] Webserver started at http://127.3.243.65:33365/ using document root <none> and password file <none>
I20260619 18:47:21.315651 7714 fs_manager.cc:359] Metadata directory not provided
I20260619 18:47:21.315762 7714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:47:21.316033 7714 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:47:21.318768 7714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894810715282-4045-0/minicluster-data/ts-0/data/instance:
uuid: "c95c585a38204cba832f54310f4a067a"
format_stamp: "Formatted at 2026-06-19 18:47:21 on dist-test-slave-m9wg"
server_key: "951c395624198c52b634af25576de708"
server_key_iv: "701f0c3b7853c9cabb1f5371005ff658"
server_key_version: "encryptionkey@0"
I20260619 18:47:21.319492 7714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894810715282-4045-0/minicluster-data/ts-0/wal/instance:
uuid: "c95c585a38204cba832f54310f4a067a"
format_stamp: "Formatted at 2026-06-19 18:47:21 on dist-test-slave-m9wg"
server_key: "951c395624198c52b634af25576de708"
server_key_iv: "701f0c3b7853c9cabb1f5371005ff658"
server_key_version: "encryptionkey@0"
I20260619 18:47:21.324121 7714 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260619 18:47:21.327204 7729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:21.328434 7714 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 18:47:21.328573 7714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894810715282-4045-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894810715282-4045-0/minicluster-data/ts-0/wal
uuid: "c95c585a38204cba832f54310f4a067a"
format_stamp: "Formatted at 2026-06-19 18:47:21 on dist-test-slave-m9wg"
server_key: "951c395624198c52b634af25576de708"
server_key_iv: "701f0c3b7853c9cabb1f5371005ff658"
server_key_version: "encryptionkey@0"
I20260619 18:47:21.328744 7714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894810715282-4045-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894810715282-4045-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894810715282-4045-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:47:21.357894 7714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:47:21.361935 7714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:47:21.362191 7714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:47:21.363102 7714 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:47:21.364401 7714 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:47:21.364473 7714 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:21.364588 7714 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:47:21.364639 7714 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:21.393565 7714 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.65:44503
I20260619 18:47:21.393601 7841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.65:44503 every 8 connection(s)
I20260619 18:47:21.394827 7714 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894810715282-4045-0/minicluster-data/ts-0/data/info.pb
I20260619 18:47:21.404278 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 7714
I20260619 18:47:21.404502 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894810715282-4045-0/minicluster-data/ts-0/wal/instance
I20260619 18:47:21.405260 4045 external_mini_cluster.cc:1428] Setting key bf36137c0e33a6789c1e850f7d47cd22
Jun 19 18:47:21 dist-test-slave-m9wg krb5kdc[7608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894841, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.65@KRBTEST.COM for kudu/127.3.243.126@KRBTEST.COM
I20260619 18:47:21.414062 7844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:21.397383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.65:54521 (local address 127.3.243.126:40059)
0619 18:47:21.397827 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:21.397832 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:21.398744 (+ 912us) server_negotiation.cc:407] Connection header received
0619 18:47:21.400561 (+ 1817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:21.400565 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:21.400641 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:21.400724 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:21.402105 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:21.402799 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:21.404330 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:21.404912 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:21.405040 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:21.408571 (+ 3531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:21.408576 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:21.408579 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:21.408611 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:21.411033 (+ 2422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:21.411855 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:21.411859 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:21.411862 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:21.411980 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:21.412438 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:21.412441 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:21.412443 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:21.412709 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:21.412818 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:21.413638 (+ 820us) server_negotiation.cc:299] Negotiation successful
0619 18:47:21.413807 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":237,"threads_started":1}
I20260619 18:47:21.415648 7845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:21.397936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:40059 (local address 127.3.243.65:54521)
0619 18:47:21.398585 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:21.398629 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:21.400307 (+ 1678us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:21.400928 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:21.400936 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:21.401421 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:47:21.401973 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:21.401991 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:21.402924 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:21.402928 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:21.404190 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:21.404206 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:21.405102 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:21.405107 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:21.405312 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:21.406151 (+ 839us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:47:21.406178 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:47:21.408368 (+ 2190us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:47:21.411162 (+ 2794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:21.411171 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:21.411189 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:21.411727 (+ 538us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:21.412071 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:21.412076 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:21.412079 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:21.412335 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:21.412824 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:47:21.412833 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:47:21.413271 (+ 438us) client_negotiation.cc:769] Sending connection context
0619 18:47:21.413562 (+ 291us) client_negotiation.cc:240] Negotiation successful
0619 18:47:21.414068 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":240,"threads_started":1}
I20260619 18:47:21.417140 7842 heartbeater.cc:347] Connected to a master server at 127.3.243.126:40059
I20260619 18:47:21.417426 7842 heartbeater.cc:464] Registering TS with master...
I20260619 18:47:21.418079 7842 heartbeater.cc:511] Master 127.3.243.126:40059 requested a full tablet report, sending...
I20260619 18:47:21.420092 7653 ts_manager.cc:194] Registered new tserver with Master: c95c585a38204cba832f54310f4a067a (127.3.243.65:44503)
I20260619 18:47:21.421485 7653 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.65@KRBTEST.COM'} at 127.3.243.65:54521
WARNING: no policy specified for kudu/127.3.243.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.66.keytab.
Entry for principal kudu/127.3.243.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.66.keytab.
I20260619 18:47:21.443266 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894810715282-4045-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894810715282-4045-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894810715282-4045-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894810715282-4045-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.66:0
--local_ip_for_outbound_sockets=127.3.243.66
--webserver_interface=127.3.243.66
--webserver_port=0
--tserver_master_addrs=127.3.243.126:40059
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.66.keytab
--principal=kudu/127.3.243.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:35045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:47:21.749275 7848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:47:21.749553 7848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:47:21.749675 7848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:47:21.756834 7848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:47:21.756945 7848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:47:21.757094 7848 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.66
I20260619 18:47:21.766544 7848 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:35045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894810715282-4045-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894810715282-4045-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.66.keytab
--principal=kudu/127.3.243.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894810715282-4045-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.243.66
--webserver_port=0
--tserver_master_addrs=127.3.243.126:40059
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.66
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894810715282-4045-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:47:21.767762 7848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:47:21.769013 7848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:47:21.781193 7856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:21.781232 7854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:21.781261 7853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:47:21.781975 7848 server_base.cc:1034] running on GCE node
I20260619 18:47:21.782477 7848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:47:21.783317 7848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:47:21.784502 7848 hybrid_clock.cc:648] HybridClock initialized: now 1781894841784457 us; error 37 us; skew 500 ppm
I20260619 18:47:21.784797 7848 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:47:21 dist-test-slave-m9wg krb5kdc[7608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894841, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:47:21.787698 7848 init.cc:377] Logged in from keytab as kudu/127.3.243.66@KRBTEST.COM (short username kudu)
I20260619 18:47:21.789111 7848 webserver.cc:460] Webserver started at http://127.3.243.66:36603/ using document root <none> and password file <none>
I20260619 18:47:21.789754 7848 fs_manager.cc:359] Metadata directory not provided
I20260619 18:47:21.789851 7848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:47:21.790092 7848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:47:21.792870 7848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894810715282-4045-0/minicluster-data/ts-1/data/instance:
uuid: "a447d860c99448638ebe8a65c1bd8c1b"
format_stamp: "Formatted at 2026-06-19 18:47:21 on dist-test-slave-m9wg"
server_key: "e9f3166130dd195deb5db3c7e45bbd38"
server_key_iv: "25ec1be01a8ab3a1b310100e2be7630e"
server_key_version: "encryptionkey@0"
I20260619 18:47:21.793558 7848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894810715282-4045-0/minicluster-data/ts-1/wal/instance:
uuid: "a447d860c99448638ebe8a65c1bd8c1b"
format_stamp: "Formatted at 2026-06-19 18:47:21 on dist-test-slave-m9wg"
server_key: "e9f3166130dd195deb5db3c7e45bbd38"
server_key_iv: "25ec1be01a8ab3a1b310100e2be7630e"
server_key_version: "encryptionkey@0"
I20260619 18:47:21.798493 7848 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 18:47:21.801718 7863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:21.802958 7848 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 18:47:21.803102 7848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894810715282-4045-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894810715282-4045-0/minicluster-data/ts-1/wal
uuid: "a447d860c99448638ebe8a65c1bd8c1b"
format_stamp: "Formatted at 2026-06-19 18:47:21 on dist-test-slave-m9wg"
server_key: "e9f3166130dd195deb5db3c7e45bbd38"
server_key_iv: "25ec1be01a8ab3a1b310100e2be7630e"
server_key_version: "encryptionkey@0"
I20260619 18:47:21.803288 7848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894810715282-4045-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894810715282-4045-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894810715282-4045-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:47:21.828855 7848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:47:21.833732 7848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:47:21.834014 7848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:47:21.835176 7848 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:47:21.836831 7848 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:47:21.836922 7848 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:21.837062 7848 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:47:21.837127 7848 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:21.869486 7848 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.66:44131
I20260619 18:47:21.869550 7975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.66:44131 every 8 connection(s)
I20260619 18:47:21.870677 7848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894810715282-4045-0/minicluster-data/ts-1/data/info.pb
I20260619 18:47:21.877306 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 7848
I20260619 18:47:21.877493 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894810715282-4045-0/minicluster-data/ts-1/wal/instance
I20260619 18:47:21.877871 4045 external_mini_cluster.cc:1428] Setting key c3d93c4b1af73377c17799edce719712
Jun 19 18:47:21 dist-test-slave-m9wg krb5kdc[7608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894841, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.66@KRBTEST.COM for kudu/127.3.243.126@KRBTEST.COM
I20260619 18:47:21.891031 7844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:21.873168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.66:58729 (local address 127.3.243.126:40059)
0619 18:47:21.873383 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:21.873388 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:21.874563 (+ 1175us) server_negotiation.cc:407] Connection header received
0619 18:47:21.876388 (+ 1825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:21.876392 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:21.876466 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:21.876559 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:21.878278 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:21.879141 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:21.880765 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:21.881384 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:21.881509 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:21.885062 (+ 3553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:21.885068 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:21.885071 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:21.885101 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:21.888068 (+ 2967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:21.888793 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:21.888796 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:21.888799 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:21.888939 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:21.889397 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:21.889401 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:21.889403 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:21.889658 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:21.889798 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:21.890485 (+ 687us) server_negotiation.cc:299] Negotiation successful
0619 18:47:21.890703 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260619 18:47:21.891968 7978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:21.873688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:40059 (local address 127.3.243.66:58729)
0619 18:47:21.874406 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:21.874450 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:21.876078 (+ 1628us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:21.876787 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:21.876810 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:21.877403 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:47:21.878088 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:21.878107 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:21.879284 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:21.879287 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:21.880614 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:21.880626 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:21.881581 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:21.881586 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:21.881784 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:21.882390 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:47:21.882413 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:47:21.884853 (+ 2440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:47:21.888230 (+ 3377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:21.888239 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:21.888256 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:21.888654 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:21.889039 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:21.889044 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:21.889048 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:21.889259 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:21.889785 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:47:21.889792 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:47:21.890129 (+ 337us) client_negotiation.cc:769] Sending connection context
0619 18:47:21.890360 (+ 231us) client_negotiation.cc:240] Negotiation successful
0619 18:47:21.890888 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":264,"threads_started":1}
I20260619 18:47:21.893342 7976 heartbeater.cc:347] Connected to a master server at 127.3.243.126:40059
I20260619 18:47:21.893682 7976 heartbeater.cc:464] Registering TS with master...
I20260619 18:47:21.894551 7976 heartbeater.cc:511] Master 127.3.243.126:40059 requested a full tablet report, sending...
I20260619 18:47:21.896112 7653 ts_manager.cc:194] Registered new tserver with Master: a447d860c99448638ebe8a65c1bd8c1b (127.3.243.66:44131)
WARNING: no policy specified for kudu/127.3.243.67@KRBTEST.COM; defaulting to no policy
I20260619 18:47:21.896986 7653 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.66@KRBTEST.COM'} at 127.3.243.66:58729
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.67.keytab.
Entry for principal kudu/127.3.243.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.67.keytab.
I20260619 18:47:21.919328 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894810715282-4045-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894810715282-4045-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894810715282-4045-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894810715282-4045-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.67:0
--local_ip_for_outbound_sockets=127.3.243.67
--webserver_interface=127.3.243.67
--webserver_port=0
--tserver_master_addrs=127.3.243.126:40059
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.67.keytab
--principal=kudu/127.3.243.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:35045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:47:22.223989 7981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:47:22.224311 7981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:47:22.224442 7981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:47:22.231679 7981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:47:22.231805 7981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:47:22.231954 7981 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.67
I20260619 18:47:22.241595 7981 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:35045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894810715282-4045-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894810715282-4045-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.67.keytab
--principal=kudu/127.3.243.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894810715282-4045-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.243.67
--webserver_port=0
--tserver_master_addrs=127.3.243.126:40059
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.67
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894810715282-4045-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:47:22.242833 7981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:47:22.244087 7981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:47:22.256417 7986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:22.256395 7987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:47:22.257028 7981 server_base.cc:1034] running on GCE node
W20260619 18:47:22.257299 7989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:47:22.257889 7981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:47:22.258788 7981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:47:22.259984 7981 hybrid_clock.cc:648] HybridClock initialized: now 1781894842259937 us; error 41 us; skew 500 ppm
I20260619 18:47:22.260313 7981 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:47:22 dist-test-slave-m9wg krb5kdc[7608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894842, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:47:22.263176 7981 init.cc:377] Logged in from keytab as kudu/127.3.243.67@KRBTEST.COM (short username kudu)
I20260619 18:47:22.264562 7981 webserver.cc:460] Webserver started at http://127.3.243.67:32873/ using document root <none> and password file <none>
I20260619 18:47:22.265185 7981 fs_manager.cc:359] Metadata directory not provided
I20260619 18:47:22.265308 7981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:47:22.265583 7981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:47:22.268366 7981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894810715282-4045-0/minicluster-data/ts-2/data/instance:
uuid: "c787b7a8f3b54d8e8fec950a746152f8"
format_stamp: "Formatted at 2026-06-19 18:47:22 on dist-test-slave-m9wg"
server_key: "a2f198a71f4e6e6d07286fe0242f2199"
server_key_iv: "dbc4699a8630d7187db3937e85f5ead8"
server_key_version: "encryptionkey@0"
I20260619 18:47:22.269065 7981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894810715282-4045-0/minicluster-data/ts-2/wal/instance:
uuid: "c787b7a8f3b54d8e8fec950a746152f8"
format_stamp: "Formatted at 2026-06-19 18:47:22 on dist-test-slave-m9wg"
server_key: "a2f198a71f4e6e6d07286fe0242f2199"
server_key_iv: "dbc4699a8630d7187db3937e85f5ead8"
server_key_version: "encryptionkey@0"
I20260619 18:47:22.273788 7981 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 18:47:22.277108 7996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:22.278450 7981 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 18:47:22.278605 7981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894810715282-4045-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894810715282-4045-0/minicluster-data/ts-2/wal
uuid: "c787b7a8f3b54d8e8fec950a746152f8"
format_stamp: "Formatted at 2026-06-19 18:47:22 on dist-test-slave-m9wg"
server_key: "a2f198a71f4e6e6d07286fe0242f2199"
server_key_iv: "dbc4699a8630d7187db3937e85f5ead8"
server_key_version: "encryptionkey@0"
I20260619 18:47:22.278798 7981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894810715282-4045-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894810715282-4045-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894810715282-4045-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:47:22.291950 7981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:47:22.295871 7981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:47:22.296118 7981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:47:22.296998 7981 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:47:22.298206 7981 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:47:22.298280 7981 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:22.298398 7981 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:47:22.298453 7981 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:22.326927 7981 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.67:43315
I20260619 18:47:22.326962 8108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.67:43315 every 8 connection(s)
I20260619 18:47:22.328133 7981 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894810715282-4045-0/minicluster-data/ts-2/data/info.pb
I20260619 18:47:22.334997 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 7981
I20260619 18:47:22.335210 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894810715282-4045-0/minicluster-data/ts-2/wal/instance
I20260619 18:47:22.335734 4045 external_mini_cluster.cc:1428] Setting key 88dbb28d356444472d0245ca0e050bb3
Jun 19 18:47:22 dist-test-slave-m9wg krb5kdc[7608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894842, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.67@KRBTEST.COM for kudu/127.3.243.126@KRBTEST.COM
I20260619 18:47:22.347106 7844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:22.330548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.67:50785 (local address 127.3.243.126:40059)
0619 18:47:22.330725 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:22.330731 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:22.332185 (+ 1454us) server_negotiation.cc:407] Connection header received
0619 18:47:22.334002 (+ 1817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:22.334007 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:22.334076 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:22.334166 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:22.335989 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:22.336894 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:22.338928 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:22.339331 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:22.339420 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:22.342470 (+ 3050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:22.342474 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:22.342478 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:22.342508 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:22.344811 (+ 2303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:22.345418 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:22.345422 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:22.345424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:22.345533 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:22.345853 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:22.345856 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:22.345858 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:22.346090 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:22.346191 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:22.346717 (+ 526us) server_negotiation.cc:299] Negotiation successful
0619 18:47:22.346877 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 18:47:22.348001 8111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:22.331052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:40059 (local address 127.3.243.67:50785)
0619 18:47:22.331987 (+ 935us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:22.332038 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:22.333723 (+ 1685us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:22.334350 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:22.334358 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:22.335008 (+ 650us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:47:22.335804 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:22.335842 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:22.337135 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:22.337140 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:22.338784 (+ 1644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:22.338800 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:22.339492 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:22.339497 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:22.339690 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:22.340283 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:47:22.340305 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:47:22.342289 (+ 1984us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:47:22.344934 (+ 2645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:22.344940 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:22.344952 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:22.345336 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:22.345606 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:22.345609 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:22.345611 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:22.345774 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:22.346173 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:47:22.346178 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:47:22.346453 (+ 275us) client_negotiation.cc:769] Sending connection context
0619 18:47:22.346661 (+ 208us) client_negotiation.cc:240] Negotiation successful
0619 18:47:22.347024 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":408,"threads_started":1}
I20260619 18:47:22.349167 8109 heartbeater.cc:347] Connected to a master server at 127.3.243.126:40059
I20260619 18:47:22.349457 8109 heartbeater.cc:464] Registering TS with master...
I20260619 18:47:22.350207 8109 heartbeater.cc:511] Master 127.3.243.126:40059 requested a full tablet report, sending...
I20260619 18:47:22.351536 7653 ts_manager.cc:194] Registered new tserver with Master: c787b7a8f3b54d8e8fec950a746152f8 (127.3.243.67:43315)
I20260619 18:47:22.352346 7653 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.67@KRBTEST.COM'} at 127.3.243.67:50785
I20260619 18:47:22.362558 4045 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 18:47:22 dist-test-slave-m9wg krb5kdc[7608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 18:47:22 dist-test-slave-m9wg krb5kdc[7608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.65@KRBTEST.COM
I20260619 18:47:22.409899 8116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:22.390807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60962 (local address 127.3.243.65:44503)
0619 18:47:22.391367 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:22.391373 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:22.391400 (+ 27us) server_negotiation.cc:407] Connection header received
0619 18:47:22.391456 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:22.391462 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:22.391663 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:22.391781 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:22.393117 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:22.394704 (+ 1587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:22.395886 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:22.396770 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:22.396903 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:22.404871 (+ 7968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:22.404880 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:22.404891 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:22.404923 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:22.407425 (+ 2502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:22.408036 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:22.408042 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:22.408044 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:22.408185 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:22.408518 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:22.408524 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:22.408527 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:22.408830 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:22.409090 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:22.409366 (+ 276us) server_negotiation.cc:299] Negotiation successful
0619 18:47:22.409599 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":278,"threads_started":1}
Jun 19 18:47:22 dist-test-slave-m9wg krb5kdc[7608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.126@KRBTEST.COM
I20260619 18:47:22.424271 7842 heartbeater.cc:503] Master 127.3.243.126:40059 was elected leader, sending a full tablet report...
I20260619 18:47:22.425971 7844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:22.413679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36712 (local address 127.3.243.126:40059)
0619 18:47:22.413872 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:22.413878 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:22.414013 (+ 135us) server_negotiation.cc:407] Connection header received
0619 18:47:22.414164 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:22.414168 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:22.414230 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:22.414330 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:22.415417 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:22.416115 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:22.417302 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:22.417690 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:22.417794 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:22.421214 (+ 3420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:22.421219 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:22.421222 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:22.421256 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:22.423542 (+ 2286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:22.424106 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:22.424112 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:22.424115 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:22.424264 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:22.424600 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:22.424605 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:22.424608 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:22.424981 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:22.425202 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:22.425485 (+ 283us) server_negotiation.cc:299] Negotiation successful
0619 18:47:22.425693 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260619 18:47:22.426409 7653 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36712
I20260619 18:47:22.427224 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 7714
I20260619 18:47:22.438656 8116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:22.427918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60972 (local address 127.3.243.65:44503)
0619 18:47:22.428065 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:22.428071 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:22.428354 (+ 283us) server_negotiation.cc:407] Connection header received
0619 18:47:22.428482 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:22.428487 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:22.428569 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:22.428644 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:22.429683 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:22.430320 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:22.431535 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:22.432032 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:22.432127 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:22.434173 (+ 2046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:22.434179 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:22.434182 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:22.434208 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:22.436504 (+ 2296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:22.437076 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:22.437082 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:22.437085 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:22.437192 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:22.437496 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:22.437502 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:22.437505 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:22.437724 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:22.437923 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:22.438195 (+ 272us) server_negotiation.cc:299] Negotiation successful
0619 18:47:22.438397 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 18:47:22.439143 7837 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:47:22.520396 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 7714
I20260619 18:47:22.533159 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 7848
Jun 19 18:47:22 dist-test-slave-m9wg krb5kdc[7608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.66@KRBTEST.COM
I20260619 18:47:22.547458 8121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:22.534024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35514 (local address 127.3.243.66:44131)
0619 18:47:22.534546 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:22.534552 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:22.534577 (+ 25us) server_negotiation.cc:407] Connection header received
0619 18:47:22.534636 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:22.534643 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:22.534863 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:22.534984 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:22.536097 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:22.537122 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:22.538278 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:22.538810 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:22.538904 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:22.542652 (+ 3748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:22.542662 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:22.542677 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:22.542708 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:22.545302 (+ 2594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:22.545840 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:22.545846 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:22.545848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:22.545959 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:22.546216 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:22.546219 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:22.546221 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:22.546523 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:22.546736 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:22.546931 (+ 195us) server_negotiation.cc:299] Negotiation successful
0619 18:47:22.547122 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":255,"threads_started":1}
I20260619 18:47:22.548795 7971 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:47:22.629266 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 7848
I20260619 18:47:22.644891 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 7981
Jun 19 18:47:22 dist-test-slave-m9wg krb5kdc[7608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.67@KRBTEST.COM
I20260619 18:47:22.660740 8123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:22.645892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57806 (local address 127.3.243.67:43315)
0619 18:47:22.646515 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:22.646522 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:22.646551 (+ 29us) server_negotiation.cc:407] Connection header received
0619 18:47:22.646611 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:22.646618 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:22.646847 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:22.646999 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:22.648299 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:22.649335 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:22.650598 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:22.651172 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:22.651309 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:22.655409 (+ 4100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:22.655421 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:22.655437 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:22.655471 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:22.658013 (+ 2542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:22.658685 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:22.658692 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:22.658694 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:22.658818 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:22.659177 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:22.659181 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:22.659183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:22.659537 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:22.659849 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:22.660125 (+ 276us) server_negotiation.cc:299] Negotiation successful
0619 18:47:22.660417 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":329,"threads_started":1}
I20260619 18:47:22.662047 8104 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:47:22.743134 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 7981
I20260619 18:47:22.755534 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 7621
I20260619 18:47:22.756776 7683 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:47:22.859082 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 7621
2026-06-19T18:47:22Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3613 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894810715282-4045-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:47:22 dist-test-slave-m9wg krb5kdc[8128](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:47:22 dist-test-slave-m9wg krb5kdc[8128](info): set up 2 sockets
Jun 19 18:47:22 dist-test-slave-m9wg krb5kdc[8128](info): commencing operation
krb5kdc: starting...
W20260619 18:47:23.856545 4045 mini_kdc.cc:121] Time spent starting KDC: real 0.960s user 0.005s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894810715282-4045-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894810715282-4045-0/krb5kdc/test-user.keytab.
Jun 19 18:47:23 dist-test-slave-m9wg krb5kdc[8128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:47:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:47:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.243.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab.
Entry for principal kudu/127.3.243.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab.
I20260619 18:47:24.039039 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894810715282-4045-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894810715282-4045-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894810715282-4045-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.126:35039
--webserver_interface=127.3.243.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab
--principal=kudu/127.3.243.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:42977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.126:35039
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:47:24.344333 8141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:47:24.344637 8141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:47:24.344787 8141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:47:24.351773 8141 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:47:24.351873 8141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:47:24.351949 8141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:47:24.352020 8141 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:47:24.352085 8141 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:47:24.361568 8141 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:42977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894810715282-4045-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894810715282-4045-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.243.126:35039
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab
--principal=kudu/127.3.243.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.126:35039
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.243.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894810715282-4045-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:47:24.362794 8141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:47:24.364058 8141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:47:24.374615 8146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:24.374641 8147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:24.374583 8149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:47:24.375238 8141 server_base.cc:1034] running on GCE node
I20260619 18:47:24.375962 8141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:47:24.377264 8141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:47:24.378453 8141 hybrid_clock.cc:648] HybridClock initialized: now 1781894844378403 us; error 44 us; skew 500 ppm
I20260619 18:47:24.378760 8141 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:47:24 dist-test-slave-m9wg krb5kdc[8128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894844, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:47:24.381527 8141 init.cc:377] Logged in from keytab as kudu/127.3.243.126@KRBTEST.COM (short username kudu)
I20260619 18:47:24.382565 8141 fs_manager.cc:359] Metadata directory not provided
I20260619 18:47:24.382648 8141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:47:24.382903 8141 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:47:24.385628 8141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894810715282-4045-0/minicluster-data/master-0/data/instance:
uuid: "8a109cd086a74a42af16432fb42273d3"
format_stamp: "Formatted at 2026-06-19 18:47:24 on dist-test-slave-m9wg"
server_key: "c0e55d593a32ec47223e8c9aa58b58a1"
server_key_iv: "bb2af5ab29fa9beb46b4ba4ee06ddfd8"
server_key_version: "encryptionkey@0"
I20260619 18:47:24.386330 8141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894810715282-4045-0/minicluster-data/master-0/wal/instance:
uuid: "8a109cd086a74a42af16432fb42273d3"
format_stamp: "Formatted at 2026-06-19 18:47:24 on dist-test-slave-m9wg"
server_key: "c0e55d593a32ec47223e8c9aa58b58a1"
server_key_iv: "bb2af5ab29fa9beb46b4ba4ee06ddfd8"
server_key_version: "encryptionkey@0"
I20260619 18:47:24.390873 8141 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260619 18:47:24.393972 8155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:24.395138 8141 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 18:47:24.395275 8141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894810715282-4045-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894810715282-4045-0/minicluster-data/master-0/wal
uuid: "8a109cd086a74a42af16432fb42273d3"
format_stamp: "Formatted at 2026-06-19 18:47:24 on dist-test-slave-m9wg"
server_key: "c0e55d593a32ec47223e8c9aa58b58a1"
server_key_iv: "bb2af5ab29fa9beb46b4ba4ee06ddfd8"
server_key_version: "encryptionkey@0"
I20260619 18:47:24.395448 8141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894810715282-4045-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894810715282-4045-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894810715282-4045-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:47:24.418699 8141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:47:24.422648 8141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:47:24.422888 8141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:47:24.438947 8141 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.126:35039
I20260619 18:47:24.438970 8206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.126:35039 every 8 connection(s)
I20260619 18:47:24.439742 8141 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
I20260619 18:47:24.443207 8207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:47:24.444504 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 8141
I20260619 18:47:24.444710 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894810715282-4045-0/minicluster-data/master-0/wal/instance
I20260619 18:47:24.445508 4045 external_mini_cluster.cc:1428] Setting key eacf77731018c66d0814a6b08fa1728b
I20260619 18:47:24.455107 8207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a109cd086a74a42af16432fb42273d3: Bootstrap starting.
I20260619 18:47:24.459019 8207 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a109cd086a74a42af16432fb42273d3: Neither blocks nor log segments found. Creating new log.
Jun 19 18:47:24 dist-test-slave-m9wg krb5kdc[8128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.126@KRBTEST.COM
I20260619 18:47:24.460175 8207 log.cc:824] T 00000000000000000000000000000000 P 8a109cd086a74a42af16432fb42273d3: Log is configured to *not* fsync() on all Append() calls
I20260619 18:47:24.463165 8207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a109cd086a74a42af16432fb42273d3: No bootstrap required, opened a new log
I20260619 18:47:24.468012 8210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:24.447482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44822 (local address 127.3.243.126:35039)
0619 18:47:24.448324 (+ 842us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:24.448336 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:24.448384 (+ 48us) server_negotiation.cc:407] Connection header received
0619 18:47:24.449998 (+ 1614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:24.450020 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:24.450414 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:24.450723 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:24.452654 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:24.453989 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:24.455683 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:24.456412 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:24.456549 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:24.461121 (+ 4572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:24.461132 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:24.461150 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:24.461189 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:24.463985 (+ 2796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:24.464701 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:24.464708 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:24.464715 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:24.464890 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:24.465269 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:24.465272 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:24.465274 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:24.465828 (+ 554us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:24.466103 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:24.466554 (+ 451us) server_negotiation.cc:299] Negotiation successful
0619 18:47:24.466966 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":363,"threads_started":1}
I20260619 18:47:24.471290 8207 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a109cd086a74a42af16432fb42273d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a109cd086a74a42af16432fb42273d3" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 35039 } }
I20260619 18:47:24.471638 8207 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8a109cd086a74a42af16432fb42273d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:47:24.471729 8207 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a109cd086a74a42af16432fb42273d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a109cd086a74a42af16432fb42273d3, State: Initialized, Role: FOLLOWER
I20260619 18:47:24.472277 8207 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a109cd086a74a42af16432fb42273d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a109cd086a74a42af16432fb42273d3" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 35039 } }
I20260619 18:47:24.472540 8207 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8a109cd086a74a42af16432fb42273d3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:47:24.472672 8207 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a109cd086a74a42af16432fb42273d3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:47:24.472872 8207 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a109cd086a74a42af16432fb42273d3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:47:24.475109 8207 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a109cd086a74a42af16432fb42273d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a109cd086a74a42af16432fb42273d3" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 35039 } }
I20260619 18:47:24.475606 8207 leader_election.cc:302] T 00000000000000000000000000000000 P 8a109cd086a74a42af16432fb42273d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a109cd086a74a42af16432fb42273d3; no voters:
I20260619 18:47:24.476181 8207 leader_election.cc:288] T 00000000000000000000000000000000 P 8a109cd086a74a42af16432fb42273d3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:47:24.476357 8212 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a109cd086a74a42af16432fb42273d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:47:24.477604 8212 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a109cd086a74a42af16432fb42273d3 [term 1 LEADER]: Becoming Leader. State: Replica: 8a109cd086a74a42af16432fb42273d3, State: Running, Role: LEADER
I20260619 18:47:24.478175 8212 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a109cd086a74a42af16432fb42273d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a109cd086a74a42af16432fb42273d3" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 35039 } }
I20260619 18:47:24.478426 8207 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a109cd086a74a42af16432fb42273d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:47:24.480603 8213 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a109cd086a74a42af16432fb42273d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8a109cd086a74a42af16432fb42273d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a109cd086a74a42af16432fb42273d3" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 35039 } } }
I20260619 18:47:24.480881 8213 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a109cd086a74a42af16432fb42273d3 [sys.catalog]: This master's current role is: LEADER
I20260619 18:47:24.481680 8219 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:47:24.481526 8214 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a109cd086a74a42af16432fb42273d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a109cd086a74a42af16432fb42273d3. Latest consensus state: current_term: 1 leader_uuid: "8a109cd086a74a42af16432fb42273d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a109cd086a74a42af16432fb42273d3" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 35039 } } }
I20260619 18:47:24.481850 8214 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a109cd086a74a42af16432fb42273d3 [sys.catalog]: This master's current role is: LEADER
I20260619 18:47:24.487412 8219 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:47:24.498049 8219 catalog_manager.cc:1347] Generated new cluster ID: 20c34f0b39b7465f82d52626791172a9
I20260619 18:47:24.498163 8219 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:47:24.510946 8219 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:47:24.512269 8219 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:47:24.517297 8219 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8a109cd086a74a42af16432fb42273d3: Generated new TSK 0
I20260619 18:47:24.518059 8219 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.243.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.65.keytab.
Entry for principal kudu/127.3.243.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.65.keytab.
I20260619 18:47:24.567251 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894810715282-4045-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894810715282-4045-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894810715282-4045-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894810715282-4045-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.65:0
--local_ip_for_outbound_sockets=127.3.243.65
--webserver_interface=127.3.243.65
--webserver_port=0
--tserver_master_addrs=127.3.243.126:35039
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.65.keytab
--principal=kudu/127.3.243.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:42977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:47:24.891531 8233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:47:24.891798 8233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:47:24.891920 8233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:47:24.899132 8233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:47:24.899231 8233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:47:24.899379 8233 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.65
I20260619 18:47:24.908805 8233 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:42977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894810715282-4045-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894810715282-4045-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.65.keytab
--principal=kudu/127.3.243.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894810715282-4045-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.243.65
--webserver_port=0
--tserver_master_addrs=127.3.243.126:35039
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.65
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894810715282-4045-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:47:24.909955 8233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:47:24.911108 8233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:47:24.922663 8238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:24.922761 8241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:24.922791 8239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:47:24.923352 8233 server_base.cc:1034] running on GCE node
I20260619 18:47:24.923856 8233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:47:24.924666 8233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:47:24.925872 8233 hybrid_clock.cc:648] HybridClock initialized: now 1781894844925820 us; error 38 us; skew 500 ppm
I20260619 18:47:24.926168 8233 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:47:24 dist-test-slave-m9wg krb5kdc[8128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894844, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:47:24.929042 8233 init.cc:377] Logged in from keytab as kudu/127.3.243.65@KRBTEST.COM (short username kudu)
I20260619 18:47:24.930032 8233 fs_manager.cc:359] Metadata directory not provided
I20260619 18:47:24.930128 8233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:47:24.930404 8233 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:47:24.933169 8233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894810715282-4045-0/minicluster-data/ts-0/data/instance:
uuid: "4e319002886f404c88cf62d784aa1fa5"
format_stamp: "Formatted at 2026-06-19 18:47:24 on dist-test-slave-m9wg"
server_key: "35abf9d21eda3f5010498d87f6421747"
server_key_iv: "ecc3e440a2808bfae3de60e845cb1f01"
server_key_version: "encryptionkey@0"
I20260619 18:47:24.933969 8233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894810715282-4045-0/minicluster-data/ts-0/wal/instance:
uuid: "4e319002886f404c88cf62d784aa1fa5"
format_stamp: "Formatted at 2026-06-19 18:47:24 on dist-test-slave-m9wg"
server_key: "35abf9d21eda3f5010498d87f6421747"
server_key_iv: "ecc3e440a2808bfae3de60e845cb1f01"
server_key_version: "encryptionkey@0"
I20260619 18:47:24.938678 8233 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260619 18:47:24.941795 8247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:24.942963 8233 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 18:47:24.943110 8233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894810715282-4045-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894810715282-4045-0/minicluster-data/ts-0/wal
uuid: "4e319002886f404c88cf62d784aa1fa5"
format_stamp: "Formatted at 2026-06-19 18:47:24 on dist-test-slave-m9wg"
server_key: "35abf9d21eda3f5010498d87f6421747"
server_key_iv: "ecc3e440a2808bfae3de60e845cb1f01"
server_key_version: "encryptionkey@0"
I20260619 18:47:24.943328 8233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894810715282-4045-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894810715282-4045-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894810715282-4045-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:47:24.970199 8233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:47:24.973963 8233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:47:24.974196 8233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:47:24.975069 8233 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:47:24.976326 8233 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:47:24.976419 8233 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:24.976518 8233 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:47:24.976591 8233 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:25.004495 8233 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.65:33291
I20260619 18:47:25.004554 8359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.65:33291 every 8 connection(s)
I20260619 18:47:25.005275 8233 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894810715282-4045-0/minicluster-data/ts-0/data/info.pb
I20260619 18:47:25.014580 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 8233
I20260619 18:47:25.014803 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894810715282-4045-0/minicluster-data/ts-0/wal/instance
I20260619 18:47:25.015554 4045 external_mini_cluster.cc:1428] Setting key 1f81d3f834f0157a3a63a7addc683d6d
Jun 19 18:47:25 dist-test-slave-m9wg krb5kdc[8128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894844, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.65@KRBTEST.COM for kudu/127.3.243.126@KRBTEST.COM
I20260619 18:47:25.023547 8362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:25.007429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.65:50477 (local address 127.3.243.126:35039)
0619 18:47:25.007834 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:25.007838 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:25.008781 (+ 943us) server_negotiation.cc:407] Connection header received
0619 18:47:25.010461 (+ 1680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:25.010465 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:25.010530 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:25.010617 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:25.011926 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:25.012680 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:25.014066 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:25.014419 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:25.014501 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:25.018114 (+ 3613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:25.018119 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:25.018123 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:25.018154 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:25.021020 (+ 2866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:25.021629 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:25.021634 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:25.021637 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:25.021775 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:25.022105 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:25.022110 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:25.022113 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:25.022415 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:25.022538 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:25.023094 (+ 556us) server_negotiation.cc:299] Negotiation successful
0619 18:47:25.023299 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":206,"threads_started":1}
I20260619 18:47:25.024686 8363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:25.007956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:35039 (local address 127.3.243.65:50477)
0619 18:47:25.008629 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:25.008669 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:25.010219 (+ 1550us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:25.010790 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:25.010798 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:25.011268 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:47:25.011804 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:25.011821 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:25.012787 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:25.012790 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:25.013959 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:25.013971 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:25.014634 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:25.014642 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:25.014920 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:25.015543 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:47:25.015578 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:47:25.017914 (+ 2336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:47:25.021138 (+ 3224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:25.021144 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:25.021155 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:25.021532 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:25.021868 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:25.021871 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:25.021874 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:25.022023 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:25.022512 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:47:25.022518 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:47:25.022779 (+ 261us) client_negotiation.cc:769] Sending connection context
0619 18:47:25.023015 (+ 236us) client_negotiation.cc:240] Negotiation successful
0619 18:47:25.023421 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":288,"threads_started":1}
I20260619 18:47:25.026028 8360 heartbeater.cc:347] Connected to a master server at 127.3.243.126:35039
I20260619 18:47:25.026278 8360 heartbeater.cc:464] Registering TS with master...
I20260619 18:47:25.026830 8360 heartbeater.cc:511] Master 127.3.243.126:35039 requested a full tablet report, sending...
I20260619 18:47:25.028605 8172 ts_manager.cc:194] Registered new tserver with Master: 4e319002886f404c88cf62d784aa1fa5 (127.3.243.65:33291)
I20260619 18:47:25.029983 8172 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.65@KRBTEST.COM'} at 127.3.243.65:50477
WARNING: no policy specified for kudu/127.3.243.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.66.keytab.
Entry for principal kudu/127.3.243.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.66.keytab.
I20260619 18:47:25.054999 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894810715282-4045-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894810715282-4045-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894810715282-4045-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894810715282-4045-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.66:0
--local_ip_for_outbound_sockets=127.3.243.66
--webserver_interface=127.3.243.66
--webserver_port=0
--tserver_master_addrs=127.3.243.126:35039
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.66.keytab
--principal=kudu/127.3.243.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:42977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:47:25.359342 8366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:47:25.359642 8366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:47:25.359789 8366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:47:25.366971 8366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:47:25.367095 8366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:47:25.367256 8366 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.66
I20260619 18:47:25.376912 8366 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:42977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894810715282-4045-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894810715282-4045-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.66.keytab
--principal=kudu/127.3.243.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894810715282-4045-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.243.66
--webserver_port=0
--tserver_master_addrs=127.3.243.126:35039
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.66
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894810715282-4045-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:47:25.378175 8366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:47:25.379490 8366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:47:25.391826 8371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:25.391747 8374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:25.392027 8372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:47:25.392777 8366 server_base.cc:1034] running on GCE node
I20260619 18:47:25.393330 8366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:47:25.394164 8366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:47:25.395349 8366 hybrid_clock.cc:648] HybridClock initialized: now 1781894845395277 us; error 52 us; skew 500 ppm
I20260619 18:47:25.395617 8366 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:47:25 dist-test-slave-m9wg krb5kdc[8128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894845, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:47:25.398526 8366 init.cc:377] Logged in from keytab as kudu/127.3.243.66@KRBTEST.COM (short username kudu)
I20260619 18:47:25.399565 8366 fs_manager.cc:359] Metadata directory not provided
I20260619 18:47:25.399667 8366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:47:25.399930 8366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:47:25.402722 8366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894810715282-4045-0/minicluster-data/ts-1/data/instance:
uuid: "dc740a16622643f99fe92a45dfe4cf88"
format_stamp: "Formatted at 2026-06-19 18:47:25 on dist-test-slave-m9wg"
server_key: "08523e32dcd840d045abac992878e846"
server_key_iv: "efc0a1b5edd6b6297255cfa3deebce47"
server_key_version: "encryptionkey@0"
I20260619 18:47:25.403522 8366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894810715282-4045-0/minicluster-data/ts-1/wal/instance:
uuid: "dc740a16622643f99fe92a45dfe4cf88"
format_stamp: "Formatted at 2026-06-19 18:47:25 on dist-test-slave-m9wg"
server_key: "08523e32dcd840d045abac992878e846"
server_key_iv: "efc0a1b5edd6b6297255cfa3deebce47"
server_key_version: "encryptionkey@0"
I20260619 18:47:25.408569 8366 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260619 18:47:25.412194 8380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:25.413548 8366 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 18:47:25.413712 8366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894810715282-4045-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894810715282-4045-0/minicluster-data/ts-1/wal
uuid: "dc740a16622643f99fe92a45dfe4cf88"
format_stamp: "Formatted at 2026-06-19 18:47:25 on dist-test-slave-m9wg"
server_key: "08523e32dcd840d045abac992878e846"
server_key_iv: "efc0a1b5edd6b6297255cfa3deebce47"
server_key_version: "encryptionkey@0"
I20260619 18:47:25.413916 8366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894810715282-4045-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894810715282-4045-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894810715282-4045-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:47:25.432430 8366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:47:25.436033 8366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:47:25.436316 8366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:47:25.437276 8366 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:47:25.438624 8366 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:47:25.438733 8366 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:25.438840 8366 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:47:25.438910 8366 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:25.473313 8366 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.66:39889
I20260619 18:47:25.473361 8492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.66:39889 every 8 connection(s)
I20260619 18:47:25.474115 8366 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894810715282-4045-0/minicluster-data/ts-1/data/info.pb
I20260619 18:47:25.481431 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 8366
I20260619 18:47:25.481594 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894810715282-4045-0/minicluster-data/ts-1/wal/instance
I20260619 18:47:25.481990 4045 external_mini_cluster.cc:1428] Setting key 22781418f6f26afa6f8186b30252c26c
Jun 19 18:47:25 dist-test-slave-m9wg krb5kdc[8128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894845, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.66@KRBTEST.COM for kudu/127.3.243.126@KRBTEST.COM
I20260619 18:47:25.493251 8362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:25.476455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.66:40937 (local address 127.3.243.126:35039)
0619 18:47:25.476625 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:25.476629 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:25.477763 (+ 1134us) server_negotiation.cc:407] Connection header received
0619 18:47:25.479575 (+ 1812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:25.479581 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:25.479657 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:25.479743 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:25.481218 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:25.481982 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:25.483821 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:25.484259 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:25.484371 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:25.487776 (+ 3405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:25.487781 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:25.487785 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:25.487823 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:25.490338 (+ 2515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:25.491140 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:25.491143 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:25.491145 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:25.491269 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:25.491678 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:25.491681 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:25.491683 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:25.491924 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:25.492027 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:25.492823 (+ 796us) server_negotiation.cc:299] Negotiation successful
0619 18:47:25.493002 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 18:47:25.494555 8495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:25.476940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:35039 (local address 127.3.243.66:40937)
0619 18:47:25.477608 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:25.477649 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:25.479313 (+ 1664us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:25.479972 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:25.479980 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:25.480519 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:47:25.481061 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:25.481078 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:25.482168 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:25.482174 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:25.483657 (+ 1483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:25.483669 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:25.484470 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:25.484478 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:25.484680 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:25.485319 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:47:25.485341 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:47:25.487566 (+ 2225us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:47:25.490470 (+ 2904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:25.490479 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:25.490496 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:25.491034 (+ 538us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:25.491365 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:25.491369 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:25.491372 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:25.491581 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:25.492034 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:47:25.492041 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:47:25.492470 (+ 429us) client_negotiation.cc:769] Sending connection context
0619 18:47:25.492759 (+ 289us) client_negotiation.cc:240] Negotiation successful
0619 18:47:25.493268 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":271,"threads_started":1}
I20260619 18:47:25.496001 8493 heartbeater.cc:347] Connected to a master server at 127.3.243.126:35039
I20260619 18:47:25.496342 8493 heartbeater.cc:464] Registering TS with master...
I20260619 18:47:25.497030 8493 heartbeater.cc:511] Master 127.3.243.126:35039 requested a full tablet report, sending...
I20260619 18:47:25.498459 8172 ts_manager.cc:194] Registered new tserver with Master: dc740a16622643f99fe92a45dfe4cf88 (127.3.243.66:39889)
I20260619 18:47:25.499287 8172 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.66@KRBTEST.COM'} at 127.3.243.66:40937
WARNING: no policy specified for kudu/127.3.243.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.67.keytab.
Entry for principal kudu/127.3.243.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.67.keytab.
I20260619 18:47:25.522918 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894810715282-4045-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894810715282-4045-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894810715282-4045-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894810715282-4045-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.67:0
--local_ip_for_outbound_sockets=127.3.243.67
--webserver_interface=127.3.243.67
--webserver_port=0
--tserver_master_addrs=127.3.243.126:35039
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.67.keytab
--principal=kudu/127.3.243.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:42977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:47:25.827577 8498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:47:25.827868 8498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:47:25.828009 8498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:47:25.835086 8498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:47:25.835217 8498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:47:25.835374 8498 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.67
I20260619 18:47:25.845031 8498 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:42977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894810715282-4045-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894810715282-4045-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.67.keytab
--principal=kudu/127.3.243.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894810715282-4045-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.243.67
--webserver_port=0
--tserver_master_addrs=127.3.243.126:35039
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.67
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894810715282-4045-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:47:25.846381 8498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:47:25.847689 8498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:47:25.859674 8506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:25.859622 8503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:25.859724 8504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:47:25.860464 8498 server_base.cc:1034] running on GCE node
I20260619 18:47:25.861011 8498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:47:25.861845 8498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:47:25.863054 8498 hybrid_clock.cc:648] HybridClock initialized: now 1781894845863004 us; error 36 us; skew 500 ppm
I20260619 18:47:25.863343 8498 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:47:25 dist-test-slave-m9wg krb5kdc[8128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894845, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:47:25.866180 8498 init.cc:377] Logged in from keytab as kudu/127.3.243.67@KRBTEST.COM (short username kudu)
I20260619 18:47:25.867235 8498 fs_manager.cc:359] Metadata directory not provided
I20260619 18:47:25.867332 8498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:47:25.867625 8498 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:47:25.870420 8498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894810715282-4045-0/minicluster-data/ts-2/data/instance:
uuid: "5043457086f14ce39382bce0d54b2704"
format_stamp: "Formatted at 2026-06-19 18:47:25 on dist-test-slave-m9wg"
server_key: "78f7709093051af8120c1f9e1b745640"
server_key_iv: "4d7904488df9266da6ac9aeda8b82bb1"
server_key_version: "encryptionkey@0"
I20260619 18:47:25.871193 8498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894810715282-4045-0/minicluster-data/ts-2/wal/instance:
uuid: "5043457086f14ce39382bce0d54b2704"
format_stamp: "Formatted at 2026-06-19 18:47:25 on dist-test-slave-m9wg"
server_key: "78f7709093051af8120c1f9e1b745640"
server_key_iv: "4d7904488df9266da6ac9aeda8b82bb1"
server_key_version: "encryptionkey@0"
I20260619 18:47:25.875895 8498 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.007s
I20260619 18:47:25.879287 8512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:25.880649 8498 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:47:25.880811 8498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894810715282-4045-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894810715282-4045-0/minicluster-data/ts-2/wal
uuid: "5043457086f14ce39382bce0d54b2704"
format_stamp: "Formatted at 2026-06-19 18:47:25 on dist-test-slave-m9wg"
server_key: "78f7709093051af8120c1f9e1b745640"
server_key_iv: "4d7904488df9266da6ac9aeda8b82bb1"
server_key_version: "encryptionkey@0"
I20260619 18:47:25.881017 8498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894810715282-4045-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894810715282-4045-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894810715282-4045-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:47:25.905123 8498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:47:25.908937 8498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:47:25.909185 8498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:47:25.910104 8498 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:47:25.911366 8498 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:47:25.911460 8498 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:25.911571 8498 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:47:25.911640 8498 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:25.940068 8498 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.67:44327
I20260619 18:47:25.940119 8624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.67:44327 every 8 connection(s)
I20260619 18:47:25.940903 8498 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894810715282-4045-0/minicluster-data/ts-2/data/info.pb
I20260619 18:47:25.947439 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 8498
I20260619 18:47:25.947638 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894810715282-4045-0/minicluster-data/ts-2/wal/instance
I20260619 18:47:25.948203 4045 external_mini_cluster.cc:1428] Setting key 52dd5abab92f30d2382635b4315e7c6a
Jun 19 18:47:25 dist-test-slave-m9wg krb5kdc[8128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894845, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.67@KRBTEST.COM for kudu/127.3.243.126@KRBTEST.COM
I20260619 18:47:25.960099 8362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:25.943166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.67:55161 (local address 127.3.243.126:35039)
0619 18:47:25.943358 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:25.943363 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:25.944494 (+ 1131us) server_negotiation.cc:407] Connection header received
0619 18:47:25.946279 (+ 1785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:25.946285 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:25.946377 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:25.946463 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:25.948336 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:25.949272 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:25.951157 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:25.951567 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:25.951658 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:25.954776 (+ 3118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:25.954781 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:25.954784 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:25.954817 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:25.957224 (+ 2407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:25.957915 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:25.957919 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:25.957921 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:25.958041 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:25.958441 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:25.958444 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:25.958446 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:25.958729 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:25.958863 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:25.959528 (+ 665us) server_negotiation.cc:299] Negotiation successful
0619 18:47:25.959758 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 18:47:25.961025 8627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:25.943683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:35039 (local address 127.3.243.67:55161)
0619 18:47:25.944359 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:25.944396 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:25.945998 (+ 1602us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:25.946694 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:25.946702 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:25.947337 (+ 635us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:47:25.948118 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:25.948144 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:25.949440 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:25.949445 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:25.951020 (+ 1575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:25.951035 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:25.951729 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:25.951734 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:25.951925 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:25.952548 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:47:25.952569 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:47:25.954580 (+ 2011us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:47:25.957353 (+ 2773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:25.957362 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:25.957378 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:25.957795 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:25.958139 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:25.958144 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:25.958148 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:25.958329 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:25.958852 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:47:25.958860 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:47:25.959186 (+ 326us) client_negotiation.cc:769] Sending connection context
0619 18:47:25.959409 (+ 223us) client_negotiation.cc:240] Negotiation successful
0619 18:47:25.959915 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":269,"threads_started":1}
I20260619 18:47:25.962378 8625 heartbeater.cc:347] Connected to a master server at 127.3.243.126:35039
I20260619 18:47:25.962706 8625 heartbeater.cc:464] Registering TS with master...
I20260619 18:47:25.963388 8625 heartbeater.cc:511] Master 127.3.243.126:35039 requested a full tablet report, sending...
I20260619 18:47:25.964737 8172 ts_manager.cc:194] Registered new tserver with Master: 5043457086f14ce39382bce0d54b2704 (127.3.243.67:44327)
I20260619 18:47:25.965504 8172 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.67@KRBTEST.COM'} at 127.3.243.67:55161
I20260619 18:47:25.975239 4045 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 18:47:25.991578 8362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:25.979686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44834 (local address 127.3.243.126:35039)
0619 18:47:25.979861 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:25.979867 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:25.980268 (+ 401us) server_negotiation.cc:407] Connection header received
0619 18:47:25.980451 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:25.980455 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:25.980525 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:25.980604 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:25.981984 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:25.982727 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:25.983969 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:25.984423 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:25.984539 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:25.986706 (+ 2167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:25.986711 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:25.986714 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:25.986747 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:25.989083 (+ 2336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:25.989691 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:25.989695 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:25.989698 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:25.989830 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:25.990173 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:25.990179 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:25.990182 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:25.990500 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:25.990734 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:25.991049 (+ 315us) server_negotiation.cc:299] Negotiation successful
0619 18:47:25.991262 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 18:47:25.995375 8172 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44834:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:47:26.002301 8172 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:47:26.032577 8360 heartbeater.cc:503] Master 127.3.243.126:35039 was elected leader, sending a full tablet report...
I20260619 18:47:26.040194 8640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:26.014790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42434 (local address 127.3.243.65:33291)
0619 18:47:26.031603 (+ 16813us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:26.031613 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:26.031647 (+ 34us) server_negotiation.cc:407] Connection header received
0619 18:47:26.031774 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:26.031783 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:26.032072 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:26.032344 (+ 272us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:47:26.032968 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:26.034587 (+ 1619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:26.037293 (+ 2706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:26.038718 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:26.038864 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:26.039091 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:26.039685 (+ 594us) server_negotiation.cc:299] Negotiation successful
0619 18:47:26.039850 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16612,"thread_start_us":17734,"threads_started":1}
I20260619 18:47:26.040213 8637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:26.014369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.65:33291 (local address 127.0.0.1:42434)
0619 18:47:26.027381 (+ 13012us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:26.027436 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:26.027631 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:26.032346 (+ 4715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:26.032353 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:26.032386 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:47:26.032751 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:26.032763 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:26.034795 (+ 2032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:26.034801 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:26.037148 (+ 2347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:26.037158 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:26.039214 (+ 2056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:26.039220 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:26.039435 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:26.039483 (+ 48us) client_negotiation.cc:769] Sending connection context
0619 18:47:26.039605 (+ 122us) client_negotiation.cc:240] Negotiation successful
0619 18:47:26.039778 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12716,"thread_start_us":12587,"threads_started":1}
I20260619 18:47:26.041963 8641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:26.014543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.66:39889 (local address 127.0.0.1:32840)
0619 18:47:26.034449 (+ 19906us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:26.034464 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:26.034577 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:26.035443 (+ 866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:26.035447 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:26.035469 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:47:26.035943 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:26.035962 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:26.037522 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:26.037527 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:26.039753 (+ 2226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:26.039765 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:26.041248 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:26.041255 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:26.041406 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:26.041436 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 18:47:26.041522 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 18:47:26.041632 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19780,"thread_start_us":20266,"threads_started":1}
I20260619 18:47:26.042574 8639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:26.014685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32840 (local address 127.3.243.66:39889)
0619 18:47:26.033051 (+ 18366us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:26.033060 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:26.034564 (+ 1504us) server_negotiation.cc:407] Connection header received
0619 18:47:26.034780 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:26.034787 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:26.035029 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:26.035213 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:47:26.036193 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:26.037399 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:26.039921 (+ 2522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:26.041051 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:26.041170 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:26.041357 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:26.042156 (+ 799us) server_negotiation.cc:299] Negotiation successful
0619 18:47:26.042325 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18204,"thread_start_us":18688,"threads_started":1}
I20260619 18:47:26.043678 8638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:26.014857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.67:44327 (local address 127.0.0.1:47840)
0619 18:47:26.035159 (+ 20302us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:26.035177 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:26.035366 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:26.036262 (+ 896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:26.036267 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:26.036291 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:47:26.036657 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:26.036666 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:26.038417 (+ 1751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:26.038422 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:26.041238 (+ 2816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:26.041253 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:26.043019 (+ 1766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:26.043025 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:26.043180 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:26.043209 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 18:47:26.043288 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 18:47:26.043397 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20129,"thread_start_us":21057,"threads_started":1}
I20260619 18:47:26.043808 8642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:26.015139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47840 (local address 127.3.243.67:44327)
0619 18:47:26.034042 (+ 18903us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:26.034050 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:26.035267 (+ 1217us) server_negotiation.cc:407] Connection header received
0619 18:47:26.035545 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:26.035553 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:26.035788 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:26.035956 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:47:26.036873 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:26.038267 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:26.041412 (+ 3145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:26.042827 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:26.042942 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:26.043152 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:26.043358 (+ 206us) server_negotiation.cc:299] Negotiation successful
0619 18:47:26.043501 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18736,"thread_start_us":18597,"threads_started":1}
I20260619 18:47:26.044212 8295 tablet_service.cc:1467] Processing CreateTablet for tablet 407a4440ba214951b9ca345e745043f0 (DEFAULT_TABLE table=test-table [id=581d3858c1d544709e8f1ccff2158e68]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:47:26.045555 8295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 407a4440ba214951b9ca345e745043f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:47:26.045912 8428 tablet_service.cc:1467] Processing CreateTablet for tablet 407a4440ba214951b9ca345e745043f0 (DEFAULT_TABLE table=test-table [id=581d3858c1d544709e8f1ccff2158e68]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:47:26.047230 8428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 407a4440ba214951b9ca345e745043f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:47:26.047183 8560 tablet_service.cc:1467] Processing CreateTablet for tablet 407a4440ba214951b9ca345e745043f0 (DEFAULT_TABLE table=test-table [id=581d3858c1d544709e8f1ccff2158e68]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:47:26.048457 8560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 407a4440ba214951b9ca345e745043f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:47:26.057806 8643 tablet_bootstrap.cc:492] T 407a4440ba214951b9ca345e745043f0 P 4e319002886f404c88cf62d784aa1fa5: Bootstrap starting.
I20260619 18:47:26.060911 8644 tablet_bootstrap.cc:492] T 407a4440ba214951b9ca345e745043f0 P dc740a16622643f99fe92a45dfe4cf88: Bootstrap starting.
I20260619 18:47:26.061234 8643 tablet_bootstrap.cc:654] T 407a4440ba214951b9ca345e745043f0 P 4e319002886f404c88cf62d784aa1fa5: Neither blocks nor log segments found. Creating new log.
I20260619 18:47:26.062494 8643 log.cc:824] T 407a4440ba214951b9ca345e745043f0 P 4e319002886f404c88cf62d784aa1fa5: Log is configured to *not* fsync() on all Append() calls
I20260619 18:47:26.063349 8645 tablet_bootstrap.cc:492] T 407a4440ba214951b9ca345e745043f0 P 5043457086f14ce39382bce0d54b2704: Bootstrap starting.
I20260619 18:47:26.064405 8644 tablet_bootstrap.cc:654] T 407a4440ba214951b9ca345e745043f0 P dc740a16622643f99fe92a45dfe4cf88: Neither blocks nor log segments found. Creating new log.
I20260619 18:47:26.065654 8644 log.cc:824] T 407a4440ba214951b9ca345e745043f0 P dc740a16622643f99fe92a45dfe4cf88: Log is configured to *not* fsync() on all Append() calls
I20260619 18:47:26.066538 8645 tablet_bootstrap.cc:654] T 407a4440ba214951b9ca345e745043f0 P 5043457086f14ce39382bce0d54b2704: Neither blocks nor log segments found. Creating new log.
I20260619 18:47:26.067664 8645 log.cc:824] T 407a4440ba214951b9ca345e745043f0 P 5043457086f14ce39382bce0d54b2704: Log is configured to *not* fsync() on all Append() calls
I20260619 18:47:26.068080 8644 tablet_bootstrap.cc:492] T 407a4440ba214951b9ca345e745043f0 P dc740a16622643f99fe92a45dfe4cf88: No bootstrap required, opened a new log
I20260619 18:47:26.068413 8643 tablet_bootstrap.cc:492] T 407a4440ba214951b9ca345e745043f0 P 4e319002886f404c88cf62d784aa1fa5: No bootstrap required, opened a new log
I20260619 18:47:26.068408 8644 ts_tablet_manager.cc:1397] T 407a4440ba214951b9ca345e745043f0 P dc740a16622643f99fe92a45dfe4cf88: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260619 18:47:26.068655 8643 ts_tablet_manager.cc:1397] T 407a4440ba214951b9ca345e745043f0 P 4e319002886f404c88cf62d784aa1fa5: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260619 18:47:26.070513 8645 tablet_bootstrap.cc:492] T 407a4440ba214951b9ca345e745043f0 P 5043457086f14ce39382bce0d54b2704: No bootstrap required, opened a new log
I20260619 18:47:26.070749 8645 ts_tablet_manager.cc:1397] T 407a4440ba214951b9ca345e745043f0 P 5043457086f14ce39382bce0d54b2704: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20260619 18:47:26.075434 8644 raft_consensus.cc:348] T 407a4440ba214951b9ca345e745043f0 P dc740a16622643f99fe92a45dfe4cf88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e319002886f404c88cf62d784aa1fa5" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 33291 } } peers { permanent_uuid: "dc740a16622643f99fe92a45dfe4cf88" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 39889 } } peers { permanent_uuid: "5043457086f14ce39382bce0d54b2704" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 44327 } }
I20260619 18:47:26.075773 8644 raft_consensus.cc:374] T 407a4440ba214951b9ca345e745043f0 P dc740a16622643f99fe92a45dfe4cf88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:47:26.075898 8644 raft_consensus.cc:729] T 407a4440ba214951b9ca345e745043f0 P dc740a16622643f99fe92a45dfe4cf88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc740a16622643f99fe92a45dfe4cf88, State: Initialized, Role: FOLLOWER
I20260619 18:47:26.076509 8644 consensus_queue.cc:260] T 407a4440ba214951b9ca345e745043f0 P dc740a16622643f99fe92a45dfe4cf88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e319002886f404c88cf62d784aa1fa5" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 33291 } } peers { permanent_uuid: "dc740a16622643f99fe92a45dfe4cf88" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 39889 } } peers { permanent_uuid: "5043457086f14ce39382bce0d54b2704" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 44327 } }
I20260619 18:47:26.077510 8493 heartbeater.cc:503] Master 127.3.243.126:35039 was elected leader, sending a full tablet report...
I20260619 18:47:26.078138 8644 ts_tablet_manager.cc:1428] T 407a4440ba214951b9ca345e745043f0 P dc740a16622643f99fe92a45dfe4cf88: Time spent starting tablet: real 0.010s user 0.000s sys 0.009s
I20260619 18:47:26.078667 8643 raft_consensus.cc:348] T 407a4440ba214951b9ca345e745043f0 P 4e319002886f404c88cf62d784aa1fa5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e319002886f404c88cf62d784aa1fa5" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 33291 } } peers { permanent_uuid: "dc740a16622643f99fe92a45dfe4cf88" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 39889 } } peers { permanent_uuid: "5043457086f14ce39382bce0d54b2704" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 44327 } }
I20260619 18:47:26.079121 8643 raft_consensus.cc:374] T 407a4440ba214951b9ca345e745043f0 P 4e319002886f404c88cf62d784aa1fa5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:47:26.079221 8643 raft_consensus.cc:729] T 407a4440ba214951b9ca345e745043f0 P 4e319002886f404c88cf62d784aa1fa5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e319002886f404c88cf62d784aa1fa5, State: Initialized, Role: FOLLOWER
I20260619 18:47:26.079787 8643 consensus_queue.cc:260] T 407a4440ba214951b9ca345e745043f0 P 4e319002886f404c88cf62d784aa1fa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e319002886f404c88cf62d784aa1fa5" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 33291 } } peers { permanent_uuid: "dc740a16622643f99fe92a45dfe4cf88" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 39889 } } peers { permanent_uuid: "5043457086f14ce39382bce0d54b2704" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 44327 } }
I20260619 18:47:26.080993 8645 raft_consensus.cc:348] T 407a4440ba214951b9ca345e745043f0 P 5043457086f14ce39382bce0d54b2704 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e319002886f404c88cf62d784aa1fa5" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 33291 } } peers { permanent_uuid: "dc740a16622643f99fe92a45dfe4cf88" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 39889 } } peers { permanent_uuid: "5043457086f14ce39382bce0d54b2704" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 44327 } }
I20260619 18:47:26.081449 8643 ts_tablet_manager.cc:1428] T 407a4440ba214951b9ca345e745043f0 P 4e319002886f404c88cf62d784aa1fa5: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260619 18:47:26.081462 8645 raft_consensus.cc:374] T 407a4440ba214951b9ca345e745043f0 P 5043457086f14ce39382bce0d54b2704 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:47:26.081573 8645 raft_consensus.cc:729] T 407a4440ba214951b9ca345e745043f0 P 5043457086f14ce39382bce0d54b2704 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5043457086f14ce39382bce0d54b2704, State: Initialized, Role: FOLLOWER
I20260619 18:47:26.082085 8645 consensus_queue.cc:260] T 407a4440ba214951b9ca345e745043f0 P 5043457086f14ce39382bce0d54b2704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e319002886f404c88cf62d784aa1fa5" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 33291 } } peers { permanent_uuid: "dc740a16622643f99fe92a45dfe4cf88" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 39889 } } peers { permanent_uuid: "5043457086f14ce39382bce0d54b2704" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 44327 } }
I20260619 18:47:26.083431 8625 heartbeater.cc:503] Master 127.3.243.126:35039 was elected leader, sending a full tablet report...
I20260619 18:47:26.085078 8645 ts_tablet_manager.cc:1428] T 407a4440ba214951b9ca345e745043f0 P 5043457086f14ce39382bce0d54b2704: Time spent starting tablet: real 0.014s user 0.011s sys 0.003s
I20260619 18:47:26.164690 8651 raft_consensus.cc:482] T 407a4440ba214951b9ca345e745043f0 P 5043457086f14ce39382bce0d54b2704 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:47:26.165004 8651 raft_consensus.cc:504] T 407a4440ba214951b9ca345e745043f0 P 5043457086f14ce39382bce0d54b2704 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e319002886f404c88cf62d784aa1fa5" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 33291 } } peers { permanent_uuid: "dc740a16622643f99fe92a45dfe4cf88" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 39889 } } peers { permanent_uuid: "5043457086f14ce39382bce0d54b2704" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 44327 } }
I20260619 18:47:26.166689 8651 leader_election.cc:288] T 407a4440ba214951b9ca345e745043f0 P 5043457086f14ce39382bce0d54b2704 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e319002886f404c88cf62d784aa1fa5 (127.3.243.65:33291), dc740a16622643f99fe92a45dfe4cf88 (127.3.243.66:39889)
I20260619 18:47:26.172107 8640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:26.167079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.67:60467 (local address 127.3.243.65:33291)
0619 18:47:26.167269 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:26.167274 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:26.167527 (+ 253us) server_negotiation.cc:407] Connection header received
0619 18:47:26.167715 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:26.167719 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:26.167781 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:26.167862 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:47:26.168293 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:26.169015 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:26.170545 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:26.171291 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:26.171441 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:26.171516 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:26.171743 (+ 227us) server_negotiation.cc:299] Negotiation successful
0619 18:47:26.171870 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 18:47:26.172119 8627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:26.167079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.65:33291 (local address 127.3.243.67:60467)
0619 18:47:26.167413 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:26.167432 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:26.167563 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:26.167893 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:26.167897 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:26.167923 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:47:26.168129 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:26.168138 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:26.169140 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:26.169143 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:26.170397 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:26.170408 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:26.171432 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:26.171437 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:26.171603 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:26.171626 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 18:47:26.171702 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 18:47:26.171830 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"spinlock_wait_cycles":48896}
I20260619 18:47:26.172822 8652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:26.167147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.66:39889 (local address 127.3.243.67:56587)
0619 18:47:26.167824 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:26.167838 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:26.168019 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:26.168356 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:26.168360 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:26.168382 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:47:26.168679 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:26.168687 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:26.169652 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:26.169656 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:26.170975 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:26.170985 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:26.172077 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:26.172083 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:26.172290 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:26.172324 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 18:47:26.172412 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 18:47:26.172525 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"spinlock_wait_cycles":21760,"thread_start_us":396,"threads_started":1}
I20260619 18:47:26.172847 8639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:26.167290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.67:56587 (local address 127.3.243.66:39889)
0619 18:47:26.167474 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:26.167479 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:26.167922 (+ 443us) server_negotiation.cc:407] Connection header received
0619 18:47:26.168173 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:26.168179 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:26.168255 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:26.168338 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:47:26.168830 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:26.169511 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:26.171120 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:26.171884 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:26.172019 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:26.172093 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:26.172456 (+ 363us) server_negotiation.cc:299] Negotiation successful
0619 18:47:26.172584 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 18:47:26.173200 8315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "407a4440ba214951b9ca345e745043f0" candidate_uuid: "5043457086f14ce39382bce0d54b2704" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e319002886f404c88cf62d784aa1fa5" is_pre_election: true
I20260619 18:47:26.173750 8315 raft_consensus.cc:2454] T 407a4440ba214951b9ca345e745043f0 P 4e319002886f404c88cf62d784aa1fa5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5043457086f14ce39382bce0d54b2704 in term 0.
I20260619 18:47:26.173884 8448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "407a4440ba214951b9ca345e745043f0" candidate_uuid: "5043457086f14ce39382bce0d54b2704" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc740a16622643f99fe92a45dfe4cf88" is_pre_election: true
I20260619 18:47:26.174296 8448 raft_consensus.cc:2454] T 407a4440ba214951b9ca345e745043f0 P dc740a16622643f99fe92a45dfe4cf88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5043457086f14ce39382bce0d54b2704 in term 0.
I20260619 18:47:26.174412 8513 leader_election.cc:302] T 407a4440ba214951b9ca345e745043f0 P 5043457086f14ce39382bce0d54b2704 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e319002886f404c88cf62d784aa1fa5, 5043457086f14ce39382bce0d54b2704; no voters:
I20260619 18:47:26.174793 8651 raft_consensus.cc:2781] T 407a4440ba214951b9ca345e745043f0 P 5043457086f14ce39382bce0d54b2704 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:47:26.174950 8651 raft_consensus.cc:482] T 407a4440ba214951b9ca345e745043f0 P 5043457086f14ce39382bce0d54b2704 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:47:26.175052 8651 raft_consensus.cc:3037] T 407a4440ba214951b9ca345e745043f0 P 5043457086f14ce39382bce0d54b2704 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:47:26.177234 8651 raft_consensus.cc:504] T 407a4440ba214951b9ca345e745043f0 P 5043457086f14ce39382bce0d54b2704 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e319002886f404c88cf62d784aa1fa5" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 33291 } } peers { permanent_uuid: "dc740a16622643f99fe92a45dfe4cf88" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 39889 } } peers { permanent_uuid: "5043457086f14ce39382bce0d54b2704" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 44327 } }
I20260619 18:47:26.177881 8651 leader_election.cc:288] T 407a4440ba214951b9ca345e745043f0 P 5043457086f14ce39382bce0d54b2704 [CANDIDATE]: Term 1 election: Requested vote from peers 4e319002886f404c88cf62d784aa1fa5 (127.3.243.65:33291), dc740a16622643f99fe92a45dfe4cf88 (127.3.243.66:39889)
I20260619 18:47:26.178171 8315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "407a4440ba214951b9ca345e745043f0" candidate_uuid: "5043457086f14ce39382bce0d54b2704" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e319002886f404c88cf62d784aa1fa5"
I20260619 18:47:26.178306 8448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "407a4440ba214951b9ca345e745043f0" candidate_uuid: "5043457086f14ce39382bce0d54b2704" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc740a16622643f99fe92a45dfe4cf88"
I20260619 18:47:26.178364 8315 raft_consensus.cc:3037] T 407a4440ba214951b9ca345e745043f0 P 4e319002886f404c88cf62d784aa1fa5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:47:26.178495 8448 raft_consensus.cc:3037] T 407a4440ba214951b9ca345e745043f0 P dc740a16622643f99fe92a45dfe4cf88 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:47:26.180606 8315 raft_consensus.cc:2454] T 407a4440ba214951b9ca345e745043f0 P 4e319002886f404c88cf62d784aa1fa5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5043457086f14ce39382bce0d54b2704 in term 1.
I20260619 18:47:26.180781 8448 raft_consensus.cc:2454] T 407a4440ba214951b9ca345e745043f0 P dc740a16622643f99fe92a45dfe4cf88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5043457086f14ce39382bce0d54b2704 in term 1.
I20260619 18:47:26.180946 8513 leader_election.cc:302] T 407a4440ba214951b9ca345e745043f0 P 5043457086f14ce39382bce0d54b2704 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e319002886f404c88cf62d784aa1fa5, 5043457086f14ce39382bce0d54b2704; no voters:
I20260619 18:47:26.181195 8651 raft_consensus.cc:2781] T 407a4440ba214951b9ca345e745043f0 P 5043457086f14ce39382bce0d54b2704 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:47:26.181602 8651 raft_consensus.cc:686] T 407a4440ba214951b9ca345e745043f0 P 5043457086f14ce39382bce0d54b2704 [term 1 LEADER]: Becoming Leader. State: Replica: 5043457086f14ce39382bce0d54b2704, State: Running, Role: LEADER
I20260619 18:47:26.182037 8651 consensus_queue.cc:237] T 407a4440ba214951b9ca345e745043f0 P 5043457086f14ce39382bce0d54b2704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e319002886f404c88cf62d784aa1fa5" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 33291 } } peers { permanent_uuid: "dc740a16622643f99fe92a45dfe4cf88" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 39889 } } peers { permanent_uuid: "5043457086f14ce39382bce0d54b2704" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 44327 } }
I20260619 18:47:26.186821 8171 catalog_manager.cc:5495] T 407a4440ba214951b9ca345e745043f0 P 5043457086f14ce39382bce0d54b2704 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5043457086f14ce39382bce0d54b2704 (127.3.243.67). New cstate: current_term: 1 leader_uuid: "5043457086f14ce39382bce0d54b2704" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e319002886f404c88cf62d784aa1fa5" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 33291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc740a16622643f99fe92a45dfe4cf88" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 39889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5043457086f14ce39382bce0d54b2704" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 44327 } health_report { overall_health: HEALTHY } } }
W20260619 18:47:26.192212 8626 tablet.cc:2369] T 407a4440ba214951b9ca345e745043f0 P 5043457086f14ce39382bce0d54b2704: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 18:47:26.225903 8494 tablet.cc:2369] T 407a4440ba214951b9ca345e745043f0 P dc740a16622643f99fe92a45dfe4cf88: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:47:26.234913 8642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:26.229977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47850 (local address 127.3.243.67:44327)
0619 18:47:26.230193 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:26.230199 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:26.230220 (+ 21us) server_negotiation.cc:407] Connection header received
0619 18:47:26.230352 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:26.230356 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:26.230431 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:26.230554 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:47:26.230908 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:26.231609 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:26.232940 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:26.233327 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:26.233435 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:26.233892 (+ 457us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:47:26.234363 (+ 471us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:47:26.234489 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:26.234585 (+ 96us) server_negotiation.cc:299] Negotiation successful
0619 18:47:26.234664 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 18:47:26.243089 8315 raft_consensus.cc:1261] T 407a4440ba214951b9ca345e745043f0 P 4e319002886f404c88cf62d784aa1fa5 [term 1 FOLLOWER]: Refusing update from remote peer 5043457086f14ce39382bce0d54b2704: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:47:26.243088 8448 raft_consensus.cc:1261] T 407a4440ba214951b9ca345e745043f0 P dc740a16622643f99fe92a45dfe4cf88 [term 1 FOLLOWER]: Refusing update from remote peer 5043457086f14ce39382bce0d54b2704: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:47:26.244166 8653 consensus_queue.cc:1035] T 407a4440ba214951b9ca345e745043f0 P 5043457086f14ce39382bce0d54b2704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc740a16622643f99fe92a45dfe4cf88" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 39889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:47:26.244534 8651 consensus_queue.cc:1035] T 407a4440ba214951b9ca345e745043f0 P 5043457086f14ce39382bce0d54b2704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e319002886f404c88cf62d784aa1fa5" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 33291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:47:26.255329 8656 mvcc.cc:205] Tried to move back new op lower bound from 7298641290201063424 to 7298641289965281280. Current Snapshot: MvccSnapshot[applied={T|T < 7298641290201063424}]
W20260619 18:47:26.257242 8361 tablet.cc:2369] T 407a4440ba214951b9ca345e745043f0 P 4e319002886f404c88cf62d784aa1fa5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:47:26.257742 8658 mvcc.cc:205] Tried to move back new op lower bound from 7298641290201063424 to 7298641289965281280. Current Snapshot: MvccSnapshot[applied={T|T < 7298641290201063424}]
I20260619 18:47:26.268532 8171 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44834:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:47:26.268787 8171 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44834:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:47:26.271883 8171 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8a109cd086a74a42af16432fb42273d3: Sending DeleteTablet for 3 replicas of tablet 407a4440ba214951b9ca345e745043f0
I20260619 18:47:26.272897 8295 tablet_service.cc:1514] Processing DeleteTablet for tablet 407a4440ba214951b9ca345e745043f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:47:26 UTC) from {username='kudu', principal='kudu/127.3.243.126@KRBTEST.COM'} at 127.0.0.1:42434
I20260619 18:47:26.272971 8428 tablet_service.cc:1514] Processing DeleteTablet for tablet 407a4440ba214951b9ca345e745043f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:47:26 UTC) from {username='kudu', principal='kudu/127.3.243.126@KRBTEST.COM'} at 127.0.0.1:32840
I20260619 18:47:26.273221 8560 tablet_service.cc:1514] Processing DeleteTablet for tablet 407a4440ba214951b9ca345e745043f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:47:26 UTC) from {username='kudu', principal='kudu/127.3.243.126@KRBTEST.COM'} at 127.0.0.1:47840
I20260619 18:47:26.276652 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 8233
I20260619 18:47:26.288710 8669 tablet_replica.cc:330] T 407a4440ba214951b9ca345e745043f0 P 5043457086f14ce39382bce0d54b2704: stopping tablet replica
I20260619 18:47:26.289359 8669 raft_consensus.cc:2229] T 407a4440ba214951b9ca345e745043f0 P 5043457086f14ce39382bce0d54b2704 [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:47:26.290145 8669 raft_consensus.cc:2258] T 407a4440ba214951b9ca345e745043f0 P 5043457086f14ce39382bce0d54b2704 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:47:26.292021 8668 tablet_replica.cc:330] T 407a4440ba214951b9ca345e745043f0 P dc740a16622643f99fe92a45dfe4cf88: stopping tablet replica
I20260619 18:47:26.292536 8668 raft_consensus.cc:2229] T 407a4440ba214951b9ca345e745043f0 P dc740a16622643f99fe92a45dfe4cf88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:47:26.293119 8668 raft_consensus.cc:2258] T 407a4440ba214951b9ca345e745043f0 P dc740a16622643f99fe92a45dfe4cf88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:47:26.293654 8667 tablet_replica.cc:330] T 407a4440ba214951b9ca345e745043f0 P 4e319002886f404c88cf62d784aa1fa5: stopping tablet replica
I20260619 18:47:26.295042 8669 ts_tablet_manager.cc:1905] T 407a4440ba214951b9ca345e745043f0 P 5043457086f14ce39382bce0d54b2704: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:47:26.295850 8668 ts_tablet_manager.cc:1905] T 407a4440ba214951b9ca345e745043f0 P dc740a16622643f99fe92a45dfe4cf88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:47:26.296371 8667 raft_consensus.cc:2229] T 407a4440ba214951b9ca345e745043f0 P 4e319002886f404c88cf62d784aa1fa5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:47:26.296938 8667 raft_consensus.cc:2258] T 407a4440ba214951b9ca345e745043f0 P 4e319002886f404c88cf62d784aa1fa5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:47:26.300361 8667 ts_tablet_manager.cc:1905] T 407a4440ba214951b9ca345e745043f0 P 4e319002886f404c88cf62d784aa1fa5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:47:26.302433 8669 ts_tablet_manager.cc:1918] T 407a4440ba214951b9ca345e745043f0 P 5043457086f14ce39382bce0d54b2704: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:47:26.302639 8669 log.cc:1196] T 407a4440ba214951b9ca345e745043f0 P 5043457086f14ce39382bce0d54b2704: Deleting WAL directory at /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894810715282-4045-0/minicluster-data/ts-2/wal/wals/407a4440ba214951b9ca345e745043f0
I20260619 18:47:26.303339 8669 ts_tablet_manager.cc:1939] T 407a4440ba214951b9ca345e745043f0 P 5043457086f14ce39382bce0d54b2704: Deleting consensus metadata
I20260619 18:47:26.304055 8668 ts_tablet_manager.cc:1918] T 407a4440ba214951b9ca345e745043f0 P dc740a16622643f99fe92a45dfe4cf88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:47:26.304287 8668 log.cc:1196] T 407a4440ba214951b9ca345e745043f0 P dc740a16622643f99fe92a45dfe4cf88: Deleting WAL directory at /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894810715282-4045-0/minicluster-data/ts-1/wal/wals/407a4440ba214951b9ca345e745043f0
I20260619 18:47:26.304816 8668 ts_tablet_manager.cc:1939] T 407a4440ba214951b9ca345e745043f0 P dc740a16622643f99fe92a45dfe4cf88: Deleting consensus metadata
I20260619 18:47:26.305207 8158 catalog_manager.cc:4841] TS 5043457086f14ce39382bce0d54b2704 (127.3.243.67:44327): tablet 407a4440ba214951b9ca345e745043f0 (table test-table [id=581d3858c1d544709e8f1ccff2158e68]) successfully deleted
Jun 19 18:47:26 dist-test-slave-m9wg krb5kdc[8128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.65@KRBTEST.COM
I20260619 18:47:26.306746 8157 catalog_manager.cc:4841] TS dc740a16622643f99fe92a45dfe4cf88 (127.3.243.66:39889): tablet 407a4440ba214951b9ca345e745043f0 (table test-table [id=581d3858c1d544709e8f1ccff2158e68]) successfully deleted
I20260619 18:47:26.311414 8667 ts_tablet_manager.cc:1918] T 407a4440ba214951b9ca345e745043f0 P 4e319002886f404c88cf62d784aa1fa5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:47:26.311686 8667 log.cc:1196] T 407a4440ba214951b9ca345e745043f0 P 4e319002886f404c88cf62d784aa1fa5: Deleting WAL directory at /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894810715282-4045-0/minicluster-data/ts-0/wal/wals/407a4440ba214951b9ca345e745043f0
I20260619 18:47:26.312503 8640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:26.278271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42438 (local address 127.3.243.65:33291)
0619 18:47:26.278463 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:26.278469 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:26.291310 (+ 12841us) server_negotiation.cc:407] Connection header received
0619 18:47:26.291519 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:26.291525 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:26.291619 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:26.291732 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:26.293669 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:26.294880 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:26.296896 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:26.297551 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:26.297652 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:26.306722 (+ 9070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:26.306741 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:26.306759 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:26.306808 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:26.309733 (+ 2925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:26.310425 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:26.310433 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:26.310435 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:26.310573 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:26.310940 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:26.310944 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:26.310946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:26.311307 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:26.311615 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:26.311912 (+ 297us) server_negotiation.cc:299] Negotiation successful
0619 18:47:26.312141 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 18:47:26.312505 8667 ts_tablet_manager.cc:1939] T 407a4440ba214951b9ca345e745043f0 P 4e319002886f404c88cf62d784aa1fa5: Deleting consensus metadata
I20260619 18:47:26.313220 8355 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:47:26.400229 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 8233
W20260619 18:47:26.421773 8156 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.243.65:33291 (error 108)
I20260619 18:47:26.422048 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 8366
I20260619 18:47:26.423985 8638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:26.423485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.65:33291 (local address 127.0.0.1:42448)
0619 18:47:26.423685 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:26.423807 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.65:33291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260619 18:47:26.424237 8156 catalog_manager.cc:4584] TS 4e319002886f404c88cf62d784aa1fa5 (127.3.243.65:33291): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 407a4440ba214951b9ca345e745043f0: Network error: Client connection negotiation failed: client connection to 127.3.243.65:33291: connect: Connection refused (error 111)
Jun 19 18:47:26 dist-test-slave-m9wg krb5kdc[8128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.66@KRBTEST.COM
I20260619 18:47:26.438553 8639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:26.423223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32854 (local address 127.3.243.66:39889)
0619 18:47:26.423403 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:26.423408 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:26.423428 (+ 20us) server_negotiation.cc:407] Connection header received
0619 18:47:26.423493 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:26.423497 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:26.423573 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:26.423663 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:26.425293 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:26.426416 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:26.427671 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:26.428339 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:26.428466 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:26.432378 (+ 3912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:26.432399 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:26.432419 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:26.432470 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:26.435912 (+ 3442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:26.436529 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:26.436538 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:26.436542 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:26.436707 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:26.437017 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:26.437022 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:26.437026 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:26.437477 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:26.437781 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:26.437936 (+ 155us) server_negotiation.cc:299] Negotiation successful
0619 18:47:26.438162 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 18:47:26.439262 8488 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:47:26.477680 8638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:26.477232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.65:33291 (local address 127.0.0.1:42450)
0619 18:47:26.477433 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:26.477525 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.65:33291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260619 18:47:26.478717 8638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:26.478383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.65:33291 (local address 127.0.0.1:42452)
0619 18:47:26.478518 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:26.478593 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.65:33291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260619 18:47:26.539700 8638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:26.539102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.65:33291 (local address 127.0.0.1:42454)
0619 18:47:26.539354 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:26.539481 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.65:33291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260619 18:47:26.540915 8638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:26.540525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.65:33291 (local address 127.0.0.1:42456)
0619 18:47:26.540672 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:26.540756 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.65:33291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260619 18:47:26.546037 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 8366
I20260619 18:47:26.565661 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 8498
Jun 19 18:47:26 dist-test-slave-m9wg krb5kdc[8128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.67@KRBTEST.COM
I20260619 18:47:26.582939 8642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:26.566409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47858 (local address 127.3.243.67:44327)
0619 18:47:26.566586 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:26.566590 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:26.566657 (+ 67us) server_negotiation.cc:407] Connection header received
0619 18:47:26.566863 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:26.566870 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:26.566971 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:26.567056 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:26.568777 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:26.569566 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:26.571313 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:26.571741 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:26.571834 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:26.577086 (+ 5252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:26.577101 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:26.577117 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:26.577149 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:26.580104 (+ 2955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:26.580805 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:26.580815 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:26.580818 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:26.581011 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:26.581328 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:26.581332 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:26.581336 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:26.581793 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:26.582110 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:26.582291 (+ 181us) server_negotiation.cc:299] Negotiation successful
0619 18:47:26.582529 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 18:47:26.583690 8620 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:47:26.622021 8638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:26.621420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.65:33291 (local address 127.0.0.1:42468)
0619 18:47:26.621711 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:26.621824 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.65:33291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138}
I20260619 18:47:26.623178 8638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:26.622708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.65:33291 (local address 127.0.0.1:42474)
0619 18:47:26.622900 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:26.623029 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.65:33291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260619 18:47:26.706915 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 8498
I20260619 18:47:26.722506 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 8141
I20260619 18:47:26.723608 8202 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:47:26.831385 8638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:26.830866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.65:33291 (local address 127.0.0.1:42490)
0619 18:47:26.831072 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:26.831214 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.65:33291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260619 18:47:26.831564 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 8141
2026-06-19T18:47:26Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (3972 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-19T18:47:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:47:26Z Disabled control of system clock
I20260619 18:47:26.889959 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894810715282-4045-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894810715282-4045-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894810715282-4045-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.126:42991
--webserver_interface=127.3.243.126
--webserver_port=0
--builtin_ntp_servers=127.3.243.84:38983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.126:42991
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 18:47:27.200078 8679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:47:27.200389 8679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:47:27.200481 8679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:47:27.207470 8679 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:47:27.207564 8679 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:47:27.207607 8679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:47:27.207660 8679 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:47:27.207701 8679 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:47:27.216581 8679 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:38983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894810715282-4045-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894810715282-4045-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.243.126:42991
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.126:42991
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.243.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894810715282-4045-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:47:27.217777 8679 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:47:27.219470 8679 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:47:27.230355 8687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:27.230393 8684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:27.230422 8685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:47:27.231024 8679 server_base.cc:1034] running on GCE node
I20260619 18:47:27.231709 8679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:47:27.233027 8679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:47:27.234220 8679 hybrid_clock.cc:648] HybridClock initialized: now 1781894847234184 us; error 26 us; skew 500 ppm
I20260619 18:47:27.234535 8679 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 18:47:27.236657 8679 webserver.cc:460] Webserver started at http://127.3.243.126:34837/ using document root <none> and password file <none>
I20260619 18:47:27.237282 8679 fs_manager.cc:359] Metadata directory not provided
I20260619 18:47:27.237386 8679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:47:27.237679 8679 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:47:27.240540 8679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894810715282-4045-0/minicluster-data/master-0/data/instance:
uuid: "d1f4ff5396fc478faef57efb46462e51"
format_stamp: "Formatted at 2026-06-19 18:47:27 on dist-test-slave-m9wg"
server_key: "05e599e670f641eda57360915cedc756"
server_key_iv: "c8c832c1e4865f1a2428fa995798bb16"
server_key_version: "encryptionkey@0"
I20260619 18:47:27.241263 8679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894810715282-4045-0/minicluster-data/master-0/wal/instance:
uuid: "d1f4ff5396fc478faef57efb46462e51"
format_stamp: "Formatted at 2026-06-19 18:47:27 on dist-test-slave-m9wg"
server_key: "05e599e670f641eda57360915cedc756"
server_key_iv: "c8c832c1e4865f1a2428fa995798bb16"
server_key_version: "encryptionkey@0"
I20260619 18:47:27.245944 8679 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 18:47:27.249071 8693 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:27.250348 8679 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:47:27.250496 8679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894810715282-4045-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894810715282-4045-0/minicluster-data/master-0/wal
uuid: "d1f4ff5396fc478faef57efb46462e51"
format_stamp: "Formatted at 2026-06-19 18:47:27 on dist-test-slave-m9wg"
server_key: "05e599e670f641eda57360915cedc756"
server_key_iv: "c8c832c1e4865f1a2428fa995798bb16"
server_key_version: "encryptionkey@0"
I20260619 18:47:27.250684 8679 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894810715282-4045-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894810715282-4045-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894810715282-4045-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:47:27.255978 8679 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:47:27.256738 8679 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:47:27.256996 8679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:47:27.273670 8679 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.126:42991
I20260619 18:47:27.273691 8744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.126:42991 every 8 connection(s)
I20260619 18:47:27.274931 8679 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
I20260619 18:47:27.276638 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 8679
I20260619 18:47:27.276844 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894810715282-4045-0/minicluster-data/master-0/wal/instance
I20260619 18:47:27.277650 4045 external_mini_cluster.cc:1428] Setting key 2fcfb3cc5adc6bc78f594abb76c7ed7c
I20260619 18:47:27.279330 8745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:47:27.285848 8748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:27.279784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40506 (local address 127.3.243.126:42991)
0619 18:47:27.280725 (+ 941us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:27.280736 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:27.280777 (+ 41us) server_negotiation.cc:407] Connection header received
0619 18:47:27.282275 (+ 1498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:27.282296 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:27.282729 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:27.283113 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:27.284192 (+ 1079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:27.284201 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:27.284217 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:47:27.284232 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:27.284313 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:47:27.284331 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:27.284552 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:27.284693 (+ 141us) server_negotiation.cc:299] Negotiation successful
0619 18:47:27.284946 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":356,"threads_started":1}
I20260619 18:47:27.291379 8745 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1f4ff5396fc478faef57efb46462e51: Bootstrap starting.
I20260619 18:47:27.294581 8745 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d1f4ff5396fc478faef57efb46462e51: Neither blocks nor log segments found. Creating new log.
I20260619 18:47:27.295634 8745 log.cc:824] T 00000000000000000000000000000000 P d1f4ff5396fc478faef57efb46462e51: Log is configured to *not* fsync() on all Append() calls
I20260619 18:47:27.298252 8745 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1f4ff5396fc478faef57efb46462e51: No bootstrap required, opened a new log
I20260619 18:47:27.305359 8745 raft_consensus.cc:348] T 00000000000000000000000000000000 P d1f4ff5396fc478faef57efb46462e51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1f4ff5396fc478faef57efb46462e51" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 42991 } }
I20260619 18:47:27.305693 8745 raft_consensus.cc:374] T 00000000000000000000000000000000 P d1f4ff5396fc478faef57efb46462e51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:47:27.305783 8745 raft_consensus.cc:729] T 00000000000000000000000000000000 P d1f4ff5396fc478faef57efb46462e51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f4ff5396fc478faef57efb46462e51, State: Initialized, Role: FOLLOWER
I20260619 18:47:27.306309 8745 consensus_queue.cc:260] T 00000000000000000000000000000000 P d1f4ff5396fc478faef57efb46462e51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1f4ff5396fc478faef57efb46462e51" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 42991 } }
I20260619 18:47:27.306515 8745 raft_consensus.cc:388] T 00000000000000000000000000000000 P d1f4ff5396fc478faef57efb46462e51 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:47:27.306634 8745 raft_consensus.cc:482] T 00000000000000000000000000000000 P d1f4ff5396fc478faef57efb46462e51 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:47:27.306803 8745 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d1f4ff5396fc478faef57efb46462e51 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:47:27.308880 8745 raft_consensus.cc:504] T 00000000000000000000000000000000 P d1f4ff5396fc478faef57efb46462e51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1f4ff5396fc478faef57efb46462e51" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 42991 } }
I20260619 18:47:27.309322 8745 leader_election.cc:302] T 00000000000000000000000000000000 P d1f4ff5396fc478faef57efb46462e51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d1f4ff5396fc478faef57efb46462e51; no voters:
I20260619 18:47:27.309849 8745 leader_election.cc:288] T 00000000000000000000000000000000 P d1f4ff5396fc478faef57efb46462e51 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:47:27.309952 8750 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d1f4ff5396fc478faef57efb46462e51 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:47:27.310870 8750 raft_consensus.cc:686] T 00000000000000000000000000000000 P d1f4ff5396fc478faef57efb46462e51 [term 1 LEADER]: Becoming Leader. State: Replica: d1f4ff5396fc478faef57efb46462e51, State: Running, Role: LEADER
I20260619 18:47:27.311395 8750 consensus_queue.cc:237] T 00000000000000000000000000000000 P d1f4ff5396fc478faef57efb46462e51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1f4ff5396fc478faef57efb46462e51" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 42991 } }
I20260619 18:47:27.311883 8745 sys_catalog.cc:564] T 00000000000000000000000000000000 P d1f4ff5396fc478faef57efb46462e51 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:47:27.313902 8751 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1f4ff5396fc478faef57efb46462e51 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d1f4ff5396fc478faef57efb46462e51" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1f4ff5396fc478faef57efb46462e51" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 42991 } } }
I20260619 18:47:27.313879 8752 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1f4ff5396fc478faef57efb46462e51 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d1f4ff5396fc478faef57efb46462e51. Latest consensus state: current_term: 1 leader_uuid: "d1f4ff5396fc478faef57efb46462e51" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1f4ff5396fc478faef57efb46462e51" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 42991 } } }
I20260619 18:47:27.314267 8751 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1f4ff5396fc478faef57efb46462e51 [sys.catalog]: This master's current role is: LEADER
I20260619 18:47:27.314265 8752 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1f4ff5396fc478faef57efb46462e51 [sys.catalog]: This master's current role is: LEADER
I20260619 18:47:27.315094 8756 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:47:27.320317 8756 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:47:27.328910 8756 catalog_manager.cc:1347] Generated new cluster ID: 04e1ec09ceb44c749d1009c0a6429ed8
I20260619 18:47:27.329003 8756 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:47:27.350323 8756 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:47:27.350841 8756 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:47:27.360744 8756 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d1f4ff5396fc478faef57efb46462e51: Generated new TSK 0
I20260619 18:47:27.361763 8756 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:47:27.369141 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894810715282-4045-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894810715282-4045-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894810715282-4045-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894810715282-4045-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.65:0
--local_ip_for_outbound_sockets=127.3.243.65
--webserver_interface=127.3.243.65
--webserver_port=0
--tserver_master_addrs=127.3.243.126:42991
--builtin_ntp_servers=127.3.243.84:38983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 18:47:27.680478 8769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:47:27.680732 8769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:47:27.680856 8769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:47:27.687824 8769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:47:27.687908 8769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:47:27.688040 8769 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.65
I20260619 18:47:27.696738 8769 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:38983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894810715282-4045-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894810715282-4045-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894810715282-4045-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.243.65
--webserver_port=0
--tserver_master_addrs=127.3.243.126:42991
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.65
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894810715282-4045-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:47:27.697914 8769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:47:27.699713 8769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:47:27.711558 8774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:27.711627 8775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:27.711598 8777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:47:27.712755 8769 server_base.cc:1034] running on GCE node
I20260619 18:47:27.713373 8769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:47:27.714221 8769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:47:27.715443 8769 hybrid_clock.cc:648] HybridClock initialized: now 1781894847715392 us; error 41 us; skew 500 ppm
I20260619 18:47:27.715758 8769 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 18:47:27.722641 8769 webserver.cc:460] Webserver started at http://127.3.243.65:43945/ using document root <none> and password file <none>
I20260619 18:47:27.723311 8769 fs_manager.cc:359] Metadata directory not provided
I20260619 18:47:27.723412 8769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:47:27.723707 8769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:47:27.726532 8769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894810715282-4045-0/minicluster-data/ts-0/data/instance:
uuid: "06baba1b3eb44ba3b71533629f6d5c9f"
format_stamp: "Formatted at 2026-06-19 18:47:27 on dist-test-slave-m9wg"
server_key: "b96da27b4cc492b4b4c306cb04e46ccb"
server_key_iv: "4b0a26f8e130bc98d4864d2d9cdfcc7c"
server_key_version: "encryptionkey@0"
I20260619 18:47:27.727306 8769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894810715282-4045-0/minicluster-data/ts-0/wal/instance:
uuid: "06baba1b3eb44ba3b71533629f6d5c9f"
format_stamp: "Formatted at 2026-06-19 18:47:27 on dist-test-slave-m9wg"
server_key: "b96da27b4cc492b4b4c306cb04e46ccb"
server_key_iv: "4b0a26f8e130bc98d4864d2d9cdfcc7c"
server_key_version: "encryptionkey@0"
I20260619 18:47:27.732283 8769 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260619 18:47:27.735538 8783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:27.736881 8769 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:47:27.737025 8769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894810715282-4045-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894810715282-4045-0/minicluster-data/ts-0/wal
uuid: "06baba1b3eb44ba3b71533629f6d5c9f"
format_stamp: "Formatted at 2026-06-19 18:47:27 on dist-test-slave-m9wg"
server_key: "b96da27b4cc492b4b4c306cb04e46ccb"
server_key_iv: "4b0a26f8e130bc98d4864d2d9cdfcc7c"
server_key_version: "encryptionkey@0"
I20260619 18:47:27.737234 8769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894810715282-4045-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894810715282-4045-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894810715282-4045-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:47:27.742838 8769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:47:27.743602 8769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:47:27.743848 8769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:47:27.744755 8769 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:47:27.746162 8769 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:47:27.746254 8769 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:27.746359 8769 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:47:27.746425 8769 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:27.775148 8769 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.65:36005
I20260619 18:47:27.775171 8895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.65:36005 every 8 connection(s)
I20260619 18:47:27.776520 8769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894810715282-4045-0/minicluster-data/ts-0/data/info.pb
I20260619 18:47:27.784090 8748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:27.778967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.65:55747 (local address 127.3.243.126:42991)
0619 18:47:27.779126 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:27.779130 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:27.780451 (+ 1321us) server_negotiation.cc:407] Connection header received
0619 18:47:27.782109 (+ 1658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:27.782115 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:27.782176 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:27.782263 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:27.783432 (+ 1169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:27.783438 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:27.783442 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:47:27.783460 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:27.783490 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:47:27.783501 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:27.783602 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:27.783768 (+ 166us) server_negotiation.cc:299] Negotiation successful
0619 18:47:27.783833 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 18:47:27.784996 8898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:27.779432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:42991 (local address 127.3.243.65:55747)
0619 18:47:27.780305 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:27.780341 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:27.781845 (+ 1504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:27.782489 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:27.782498 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:27.782588 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:47:27.783157 (+ 569us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:47:27.783159 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:47:27.783194 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:47:27.783274 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:47:27.783581 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:47:27.783591 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:47:27.783598 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 18:47:27.783848 (+ 250us) client_negotiation.cc:240] Negotiation successful
0619 18:47:27.784086 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":241,"threads_started":1}
I20260619 18:47:27.786322 8896 heartbeater.cc:347] Connected to a master server at 127.3.243.126:42991
I20260619 18:47:27.786561 8896 heartbeater.cc:464] Registering TS with master...
I20260619 18:47:27.786808 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 8769
I20260619 18:47:27.786959 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894810715282-4045-0/minicluster-data/ts-0/wal/instance
I20260619 18:47:27.787110 8896 heartbeater.cc:511] Master 127.3.243.126:42991 requested a full tablet report, sending...
I20260619 18:47:27.787596 4045 external_mini_cluster.cc:1428] Setting key 9347885166eeb89e9ee92ce12ece46e1
I20260619 18:47:27.789160 8710 ts_manager.cc:194] Registered new tserver with Master: 06baba1b3eb44ba3b71533629f6d5c9f (127.3.243.65:36005)
I20260619 18:47:27.790469 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894810715282-4045-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894810715282-4045-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894810715282-4045-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894810715282-4045-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.66:0
--local_ip_for_outbound_sockets=127.3.243.66
--webserver_interface=127.3.243.66
--webserver_port=0
--tserver_master_addrs=127.3.243.126:42991
--builtin_ntp_servers=127.3.243.84:38983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 18:47:28.090345 8899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:47:28.090619 8899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:47:28.090718 8899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:47:28.097762 8899 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:47:28.097860 8899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:47:28.097976 8899 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.66
I20260619 18:47:28.106932 8899 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:38983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894810715282-4045-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894810715282-4045-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894810715282-4045-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.243.66
--webserver_port=0
--tserver_master_addrs=127.3.243.126:42991
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.66
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894810715282-4045-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:47:28.108093 8899 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:47:28.109799 8899 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:47:28.121941 8904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:28.121901 8905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:28.121977 8907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:47:28.122648 8899 server_base.cc:1034] running on GCE node
I20260619 18:47:28.123183 8899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:47:28.124022 8899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:47:28.125206 8899 hybrid_clock.cc:648] HybridClock initialized: now 1781894848125155 us; error 44 us; skew 500 ppm
I20260619 18:47:28.125504 8899 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 18:47:28.127527 8899 webserver.cc:460] Webserver started at http://127.3.243.66:38659/ using document root <none> and password file <none>
I20260619 18:47:28.128176 8899 fs_manager.cc:359] Metadata directory not provided
I20260619 18:47:28.128283 8899 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:47:28.128584 8899 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:47:28.131254 8899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894810715282-4045-0/minicluster-data/ts-1/data/instance:
uuid: "1681097497944bd6a6b64c9620d806eb"
format_stamp: "Formatted at 2026-06-19 18:47:28 on dist-test-slave-m9wg"
server_key: "a7cd7674eb17baded72110932a4f8587"
server_key_iv: "2c2453af825863551a81f84e5053bf0e"
server_key_version: "encryptionkey@0"
I20260619 18:47:28.132017 8899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894810715282-4045-0/minicluster-data/ts-1/wal/instance:
uuid: "1681097497944bd6a6b64c9620d806eb"
format_stamp: "Formatted at 2026-06-19 18:47:28 on dist-test-slave-m9wg"
server_key: "a7cd7674eb17baded72110932a4f8587"
server_key_iv: "2c2453af825863551a81f84e5053bf0e"
server_key_version: "encryptionkey@0"
I20260619 18:47:28.136970 8899 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260619 18:47:28.140354 8913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:28.141875 8899 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 18:47:28.142004 8899 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894810715282-4045-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894810715282-4045-0/minicluster-data/ts-1/wal
uuid: "1681097497944bd6a6b64c9620d806eb"
format_stamp: "Formatted at 2026-06-19 18:47:28 on dist-test-slave-m9wg"
server_key: "a7cd7674eb17baded72110932a4f8587"
server_key_iv: "2c2453af825863551a81f84e5053bf0e"
server_key_version: "encryptionkey@0"
I20260619 18:47:28.142197 8899 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894810715282-4045-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894810715282-4045-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894810715282-4045-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:47:28.147648 8899 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:47:28.148486 8899 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:47:28.148721 8899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:47:28.149559 8899 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:47:28.150887 8899 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:47:28.150969 8899 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:28.151050 8899 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:47:28.151101 8899 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:28.180083 8899 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.66:37997
I20260619 18:47:28.180122 9025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.66:37997 every 8 connection(s)
I20260619 18:47:28.181447 8899 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894810715282-4045-0/minicluster-data/ts-1/data/info.pb
I20260619 18:47:28.185549 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 8899
I20260619 18:47:28.185693 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894810715282-4045-0/minicluster-data/ts-1/wal/instance
I20260619 18:47:28.186125 4045 external_mini_cluster.cc:1428] Setting key 8de75c5ec13d90f4fd0b3ab90065afad
I20260619 18:47:28.188755 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894810715282-4045-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894810715282-4045-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894810715282-4045-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894810715282-4045-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.67:0
--local_ip_for_outbound_sockets=127.3.243.67
--webserver_interface=127.3.243.67
--webserver_port=0
--tserver_master_addrs=127.3.243.126:42991
--builtin_ntp_servers=127.3.243.84:38983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260619 18:47:28.189381 8748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:28.183807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.66:40361 (local address 127.3.243.126:42991)
0619 18:47:28.184025 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:28.184031 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:28.185480 (+ 1449us) server_negotiation.cc:407] Connection header received
0619 18:47:28.187226 (+ 1746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:28.187231 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:28.187296 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:28.187389 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:28.188682 (+ 1293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:28.188687 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:28.188692 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:47:28.188711 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:28.188740 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:47:28.188751 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:28.188844 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:28.189046 (+ 202us) server_negotiation.cc:299] Negotiation successful
0619 18:47:28.189108 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 18:47:28.190379 9028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:28.184446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:42991 (local address 127.3.243.66:40361)
0619 18:47:28.185337 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:28.185372 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:28.186971 (+ 1599us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:28.187620 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:28.187631 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:28.187754 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:47:28.188415 (+ 661us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:47:28.188417 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:47:28.188454 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:47:28.188534 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:47:28.188840 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:47:28.188850 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:47:28.188858 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 18:47:28.189116 (+ 258us) client_negotiation.cc:240] Negotiation successful
0619 18:47:28.189360 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":248,"threads_started":1}
I20260619 18:47:28.192770 9026 heartbeater.cc:347] Connected to a master server at 127.3.243.126:42991
I20260619 18:47:28.193109 9026 heartbeater.cc:464] Registering TS with master...
I20260619 18:47:28.193661 9026 heartbeater.cc:511] Master 127.3.243.126:42991 requested a full tablet report, sending...
I20260619 18:47:28.194926 8710 ts_manager.cc:194] Registered new tserver with Master: 1681097497944bd6a6b64c9620d806eb (127.3.243.66:37997)
W20260619 18:47:28.498780 9029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:47:28.499060 9029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:47:28.499199 9029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:47:28.506248 9029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:47:28.506347 9029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:47:28.506496 9029 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.67
I20260619 18:47:28.515411 9029 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:38983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894810715282-4045-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894810715282-4045-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894810715282-4045-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.243.67
--webserver_port=0
--tserver_master_addrs=127.3.243.126:42991
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.67
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894810715282-4045-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:47:28.516691 9029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:47:28.518388 9029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:47:28.530540 9034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:28.530608 9037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:28.530583 9035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:47:28.531292 9029 server_base.cc:1034] running on GCE node
I20260619 18:47:28.531857 9029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:47:28.532788 9029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:47:28.533993 9029 hybrid_clock.cc:648] HybridClock initialized: now 1781894848533934 us; error 52 us; skew 500 ppm
I20260619 18:47:28.534336 9029 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 18:47:28.536430 9029 webserver.cc:460] Webserver started at http://127.3.243.67:37635/ using document root <none> and password file <none>
I20260619 18:47:28.537081 9029 fs_manager.cc:359] Metadata directory not provided
I20260619 18:47:28.537189 9029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:47:28.537477 9029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:47:28.540247 9029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894810715282-4045-0/minicluster-data/ts-2/data/instance:
uuid: "2468bf4f7fd7442c9020a9624c3d5fff"
format_stamp: "Formatted at 2026-06-19 18:47:28 on dist-test-slave-m9wg"
server_key: "b5989ec70413aab946c38dce6a3cc7b3"
server_key_iv: "a71fb5e954aa285ca766381a794b7773"
server_key_version: "encryptionkey@0"
I20260619 18:47:28.541069 9029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894810715282-4045-0/minicluster-data/ts-2/wal/instance:
uuid: "2468bf4f7fd7442c9020a9624c3d5fff"
format_stamp: "Formatted at 2026-06-19 18:47:28 on dist-test-slave-m9wg"
server_key: "b5989ec70413aab946c38dce6a3cc7b3"
server_key_iv: "a71fb5e954aa285ca766381a794b7773"
server_key_version: "encryptionkey@0"
I20260619 18:47:28.546110 9029 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 18:47:28.549322 9043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:28.550536 9029 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:47:28.550678 9029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894810715282-4045-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894810715282-4045-0/minicluster-data/ts-2/wal
uuid: "2468bf4f7fd7442c9020a9624c3d5fff"
format_stamp: "Formatted at 2026-06-19 18:47:28 on dist-test-slave-m9wg"
server_key: "b5989ec70413aab946c38dce6a3cc7b3"
server_key_iv: "a71fb5e954aa285ca766381a794b7773"
server_key_version: "encryptionkey@0"
I20260619 18:47:28.550876 9029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894810715282-4045-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894810715282-4045-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894810715282-4045-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:47:28.556033 9029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:47:28.556767 9029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:47:28.556988 9029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:47:28.557793 9029 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:47:28.559023 9029 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:47:28.559113 9029 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:28.559216 9029 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:47:28.559283 9029 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:28.587528 9029 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.67:43533
I20260619 18:47:28.587579 9155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.67:43533 every 8 connection(s)
I20260619 18:47:28.588781 9029 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894810715282-4045-0/minicluster-data/ts-2/data/info.pb
I20260619 18:47:28.595809 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 9029
I20260619 18:47:28.595957 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894810715282-4045-0/minicluster-data/ts-2/wal/instance
I20260619 18:47:28.596278 8748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:28.591027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.67:43191 (local address 127.3.243.126:42991)
0619 18:47:28.591193 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:28.591197 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:28.592563 (+ 1366us) server_negotiation.cc:407] Connection header received
0619 18:47:28.594190 (+ 1627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:28.594196 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:28.594262 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:28.594359 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:28.595535 (+ 1176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:28.595541 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:28.595545 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:47:28.595566 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:28.595598 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:47:28.595610 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:28.595724 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:28.595894 (+ 170us) server_negotiation.cc:299] Negotiation successful
0619 18:47:28.595963 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 18:47:28.596503 4045 external_mini_cluster.cc:1428] Setting key 9fb2b4ed2e3980936ce9a7e44016ed99
I20260619 18:47:28.597169 9158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:28.591499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:42991 (local address 127.3.243.67:43191)
0619 18:47:28.592409 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:28.592451 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:28.593932 (+ 1481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:28.594570 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:28.594578 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:28.594666 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:47:28.595247 (+ 581us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:47:28.595249 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:47:28.595284 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:47:28.595369 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:47:28.595692 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:47:28.595702 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:47:28.595709 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 18:47:28.595962 (+ 253us) client_negotiation.cc:240] Negotiation successful
0619 18:47:28.596246 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":260,"threads_started":1}
I20260619 18:47:28.598384 9156 heartbeater.cc:347] Connected to a master server at 127.3.243.126:42991
I20260619 18:47:28.598667 9156 heartbeater.cc:464] Registering TS with master...
I20260619 18:47:28.599229 9156 heartbeater.cc:511] Master 127.3.243.126:42991 requested a full tablet report, sending...
I20260619 18:47:28.600518 8710 ts_manager.cc:194] Registered new tserver with Master: 2468bf4f7fd7442c9020a9624c3d5fff (127.3.243.67:43533)
I20260619 18:47:28.611497 4045 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 18:47:28.617970 8748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:28.615670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40508 (local address 127.3.243.126:42991)
0619 18:47:28.615844 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:28.615850 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:28.616372 (+ 522us) server_negotiation.cc:407] Connection header received
0619 18:47:28.616527 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:28.616531 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:28.616583 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:28.616653 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:28.617424 (+ 771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:28.617428 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:28.617431 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:47:28.617445 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:28.617479 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:47:28.617494 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:28.617595 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:28.617652 (+ 57us) server_negotiation.cc:299] Negotiation successful
0619 18:47:28.617695 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 18:47:28.621558 8710 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40508:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:47:28.628191 8710 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:47:28.651604 9168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:28.641216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.67:43533 (local address 127.0.0.1:39786)
0619 18:47:28.642444 (+ 1228us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:28.642488 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:28.642641 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:28.649435 (+ 6794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:28.649445 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:28.649561 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:47:28.650414 (+ 853us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:47:28.650419 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:47:28.650455 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:47:28.650546 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:47:28.651031 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:47:28.651040 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:47:28.651046 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 18:47:28.651182 (+ 136us) client_negotiation.cc:240] Negotiation successful
0619 18:47:28.651254 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":924,"thread_start_us":1181,"threads_started":1}
I20260619 18:47:28.652024 9171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:28.642117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39786 (local address 127.3.243.67:43533)
0619 18:47:28.648748 (+ 6631us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:28.648760 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:28.648802 (+ 42us) server_negotiation.cc:407] Connection header received
0619 18:47:28.648885 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:28.648894 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:28.649182 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:28.649375 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:28.650785 (+ 1410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:28.650796 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:28.650813 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:47:28.650835 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:28.650902 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:47:28.650924 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:28.651095 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:28.651190 (+ 95us) server_negotiation.cc:299] Negotiation successful
0619 18:47:28.651296 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6422,"thread_start_us":6230,"threads_started":1}
I20260619 18:47:28.654605 9168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:28.641452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.65:36005 (local address 127.0.0.1:40372)
0619 18:47:28.651828 (+ 10376us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:28.651844 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:28.651924 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:28.652509 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:28.652514 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:28.652562 (+ 48us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:47:28.653789 (+ 1227us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:47:28.653792 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:47:28.653805 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:47:28.653840 (+ 35us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:47:28.654186 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:47:28.654191 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:47:28.654195 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 18:47:28.654283 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 18:47:28.654344 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10278,"thread_start_us":11056,"threads_started":1}
I20260619 18:47:28.654796 9170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:28.641704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40372 (local address 127.3.243.65:36005)
0619 18:47:28.650865 (+ 9161us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:28.650872 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:28.652021 (+ 1149us) server_negotiation.cc:407] Connection header received
0619 18:47:28.652100 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:28.652108 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:28.652357 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:28.652490 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:28.654000 (+ 1510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:28.654007 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:28.654017 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:47:28.654038 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:28.654082 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:47:28.654097 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:28.654208 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:28.654315 (+ 107us) server_negotiation.cc:299] Negotiation successful
0619 18:47:28.654407 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9014,"thread_start_us":8900,"threads_started":1}
I20260619 18:47:28.656076 9091 tablet_service.cc:1467] Processing CreateTablet for tablet 3df4b1fce6b64155b135ebee03bb104f (DEFAULT_TABLE table=test-table [id=e021131c5ea0460a90d8d5a42b101875]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:47:28.657184 9168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:28.652873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.66:37997 (local address 127.0.0.1:50824)
0619 18:47:28.654798 (+ 1925us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:28.654815 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:28.654902 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:28.655600 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:28.655604 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:28.655621 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:47:28.656073 (+ 452us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:47:28.656075 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:47:28.656087 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:47:28.656134 (+ 47us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:47:28.656820 (+ 686us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:47:28.656826 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:47:28.656832 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 18:47:28.656959 (+ 127us) client_negotiation.cc:240] Negotiation successful
0619 18:47:28.656998 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1842}
I20260619 18:47:28.657359 9172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:28.653217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50824 (local address 127.3.243.66:37997)
0619 18:47:28.653827 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:28.653835 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:28.654999 (+ 1164us) server_negotiation.cc:407] Connection header received
0619 18:47:28.655075 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:28.655160 (+ 85us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:28.655453 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:28.655603 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:28.656581 (+ 978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:28.656588 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:28.656599 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:47:28.656620 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:28.656666 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:47:28.656684 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:28.656860 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:28.656965 (+ 105us) server_negotiation.cc:299] Negotiation successful
0619 18:47:28.657051 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":315,"threads_started":1}
I20260619 18:47:28.657657 9091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3df4b1fce6b64155b135ebee03bb104f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:47:28.664363 8961 tablet_service.cc:1467] Processing CreateTablet for tablet 3df4b1fce6b64155b135ebee03bb104f (DEFAULT_TABLE table=test-table [id=e021131c5ea0460a90d8d5a42b101875]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:47:28.665323 8831 tablet_service.cc:1467] Processing CreateTablet for tablet 3df4b1fce6b64155b135ebee03bb104f (DEFAULT_TABLE table=test-table [id=e021131c5ea0460a90d8d5a42b101875]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:47:28.665782 8961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3df4b1fce6b64155b135ebee03bb104f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:47:28.666324 8831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3df4b1fce6b64155b135ebee03bb104f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:47:28.670514 9173 tablet_bootstrap.cc:492] T 3df4b1fce6b64155b135ebee03bb104f P 2468bf4f7fd7442c9020a9624c3d5fff: Bootstrap starting.
I20260619 18:47:28.674437 9173 tablet_bootstrap.cc:654] T 3df4b1fce6b64155b135ebee03bb104f P 2468bf4f7fd7442c9020a9624c3d5fff: Neither blocks nor log segments found. Creating new log.
I20260619 18:47:28.675606 9173 log.cc:824] T 3df4b1fce6b64155b135ebee03bb104f P 2468bf4f7fd7442c9020a9624c3d5fff: Log is configured to *not* fsync() on all Append() calls
I20260619 18:47:28.678620 9175 tablet_bootstrap.cc:492] T 3df4b1fce6b64155b135ebee03bb104f P 1681097497944bd6a6b64c9620d806eb: Bootstrap starting.
I20260619 18:47:28.679510 9173 tablet_bootstrap.cc:492] T 3df4b1fce6b64155b135ebee03bb104f P 2468bf4f7fd7442c9020a9624c3d5fff: No bootstrap required, opened a new log
I20260619 18:47:28.679791 9173 ts_tablet_manager.cc:1397] T 3df4b1fce6b64155b135ebee03bb104f P 2468bf4f7fd7442c9020a9624c3d5fff: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260619 18:47:28.681511 9175 tablet_bootstrap.cc:654] T 3df4b1fce6b64155b135ebee03bb104f P 1681097497944bd6a6b64c9620d806eb: Neither blocks nor log segments found. Creating new log.
I20260619 18:47:28.682282 9176 tablet_bootstrap.cc:492] T 3df4b1fce6b64155b135ebee03bb104f P 06baba1b3eb44ba3b71533629f6d5c9f: Bootstrap starting.
I20260619 18:47:28.683805 9175 log.cc:824] T 3df4b1fce6b64155b135ebee03bb104f P 1681097497944bd6a6b64c9620d806eb: Log is configured to *not* fsync() on all Append() calls
I20260619 18:47:28.685214 9176 tablet_bootstrap.cc:654] T 3df4b1fce6b64155b135ebee03bb104f P 06baba1b3eb44ba3b71533629f6d5c9f: Neither blocks nor log segments found. Creating new log.
I20260619 18:47:28.686082 9176 log.cc:824] T 3df4b1fce6b64155b135ebee03bb104f P 06baba1b3eb44ba3b71533629f6d5c9f: Log is configured to *not* fsync() on all Append() calls
I20260619 18:47:28.686654 9175 tablet_bootstrap.cc:492] T 3df4b1fce6b64155b135ebee03bb104f P 1681097497944bd6a6b64c9620d806eb: No bootstrap required, opened a new log
I20260619 18:47:28.686905 9175 ts_tablet_manager.cc:1397] T 3df4b1fce6b64155b135ebee03bb104f P 1681097497944bd6a6b64c9620d806eb: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.003s
I20260619 18:47:28.688310 9176 tablet_bootstrap.cc:492] T 3df4b1fce6b64155b135ebee03bb104f P 06baba1b3eb44ba3b71533629f6d5c9f: No bootstrap required, opened a new log
I20260619 18:47:28.688520 9176 ts_tablet_manager.cc:1397] T 3df4b1fce6b64155b135ebee03bb104f P 06baba1b3eb44ba3b71533629f6d5c9f: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.001s
I20260619 18:47:28.689257 9173 raft_consensus.cc:348] T 3df4b1fce6b64155b135ebee03bb104f P 2468bf4f7fd7442c9020a9624c3d5fff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2468bf4f7fd7442c9020a9624c3d5fff" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 43533 } } peers { permanent_uuid: "06baba1b3eb44ba3b71533629f6d5c9f" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 36005 } } peers { permanent_uuid: "1681097497944bd6a6b64c9620d806eb" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 37997 } }
I20260619 18:47:28.689670 9173 raft_consensus.cc:374] T 3df4b1fce6b64155b135ebee03bb104f P 2468bf4f7fd7442c9020a9624c3d5fff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:47:28.689757 9173 raft_consensus.cc:729] T 3df4b1fce6b64155b135ebee03bb104f P 2468bf4f7fd7442c9020a9624c3d5fff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2468bf4f7fd7442c9020a9624c3d5fff, State: Initialized, Role: FOLLOWER
I20260619 18:47:28.690423 9173 consensus_queue.cc:260] T 3df4b1fce6b64155b135ebee03bb104f P 2468bf4f7fd7442c9020a9624c3d5fff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2468bf4f7fd7442c9020a9624c3d5fff" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 43533 } } peers { permanent_uuid: "06baba1b3eb44ba3b71533629f6d5c9f" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 36005 } } peers { permanent_uuid: "1681097497944bd6a6b64c9620d806eb" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 37997 } }
I20260619 18:47:28.691745 9156 heartbeater.cc:503] Master 127.3.243.126:42991 was elected leader, sending a full tablet report...
I20260619 18:47:28.692364 9173 ts_tablet_manager.cc:1428] T 3df4b1fce6b64155b135ebee03bb104f P 2468bf4f7fd7442c9020a9624c3d5fff: Time spent starting tablet: real 0.012s user 0.014s sys 0.000s
I20260619 18:47:28.696244 9175 raft_consensus.cc:348] T 3df4b1fce6b64155b135ebee03bb104f P 1681097497944bd6a6b64c9620d806eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2468bf4f7fd7442c9020a9624c3d5fff" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 43533 } } peers { permanent_uuid: "06baba1b3eb44ba3b71533629f6d5c9f" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 36005 } } peers { permanent_uuid: "1681097497944bd6a6b64c9620d806eb" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 37997 } }
I20260619 18:47:28.696655 9175 raft_consensus.cc:374] T 3df4b1fce6b64155b135ebee03bb104f P 1681097497944bd6a6b64c9620d806eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:47:28.696760 9175 raft_consensus.cc:729] T 3df4b1fce6b64155b135ebee03bb104f P 1681097497944bd6a6b64c9620d806eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1681097497944bd6a6b64c9620d806eb, State: Initialized, Role: FOLLOWER
I20260619 18:47:28.696810 9176 raft_consensus.cc:348] T 3df4b1fce6b64155b135ebee03bb104f P 06baba1b3eb44ba3b71533629f6d5c9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2468bf4f7fd7442c9020a9624c3d5fff" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 43533 } } peers { permanent_uuid: "06baba1b3eb44ba3b71533629f6d5c9f" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 36005 } } peers { permanent_uuid: "1681097497944bd6a6b64c9620d806eb" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 37997 } }
I20260619 18:47:28.697185 9176 raft_consensus.cc:374] T 3df4b1fce6b64155b135ebee03bb104f P 06baba1b3eb44ba3b71533629f6d5c9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:47:28.697278 9176 raft_consensus.cc:729] T 3df4b1fce6b64155b135ebee03bb104f P 06baba1b3eb44ba3b71533629f6d5c9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06baba1b3eb44ba3b71533629f6d5c9f, State: Initialized, Role: FOLLOWER
I20260619 18:47:28.697373 9175 consensus_queue.cc:260] T 3df4b1fce6b64155b135ebee03bb104f P 1681097497944bd6a6b64c9620d806eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2468bf4f7fd7442c9020a9624c3d5fff" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 43533 } } peers { permanent_uuid: "06baba1b3eb44ba3b71533629f6d5c9f" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 36005 } } peers { permanent_uuid: "1681097497944bd6a6b64c9620d806eb" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 37997 } }
I20260619 18:47:28.697942 9176 consensus_queue.cc:260] T 3df4b1fce6b64155b135ebee03bb104f P 06baba1b3eb44ba3b71533629f6d5c9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2468bf4f7fd7442c9020a9624c3d5fff" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 43533 } } peers { permanent_uuid: "06baba1b3eb44ba3b71533629f6d5c9f" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 36005 } } peers { permanent_uuid: "1681097497944bd6a6b64c9620d806eb" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 37997 } }
I20260619 18:47:28.698529 9026 heartbeater.cc:503] Master 127.3.243.126:42991 was elected leader, sending a full tablet report...
I20260619 18:47:28.699153 9175 ts_tablet_manager.cc:1428] T 3df4b1fce6b64155b135ebee03bb104f P 1681097497944bd6a6b64c9620d806eb: Time spent starting tablet: real 0.012s user 0.011s sys 0.001s
I20260619 18:47:28.699520 8896 heartbeater.cc:503] Master 127.3.243.126:42991 was elected leader, sending a full tablet report...
I20260619 18:47:28.700018 9176 ts_tablet_manager.cc:1428] T 3df4b1fce6b64155b135ebee03bb104f P 06baba1b3eb44ba3b71533629f6d5c9f: Time spent starting tablet: real 0.011s user 0.008s sys 0.003s
I20260619 18:47:28.720577 9179 raft_consensus.cc:482] T 3df4b1fce6b64155b135ebee03bb104f P 2468bf4f7fd7442c9020a9624c3d5fff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:47:28.720868 9179 raft_consensus.cc:504] T 3df4b1fce6b64155b135ebee03bb104f P 2468bf4f7fd7442c9020a9624c3d5fff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2468bf4f7fd7442c9020a9624c3d5fff" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 43533 } } peers { permanent_uuid: "06baba1b3eb44ba3b71533629f6d5c9f" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 36005 } } peers { permanent_uuid: "1681097497944bd6a6b64c9620d806eb" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 37997 } }
I20260619 18:47:28.722568 9179 leader_election.cc:288] T 3df4b1fce6b64155b135ebee03bb104f P 2468bf4f7fd7442c9020a9624c3d5fff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06baba1b3eb44ba3b71533629f6d5c9f (127.3.243.65:36005), 1681097497944bd6a6b64c9620d806eb (127.3.243.66:37997)
I20260619 18:47:28.724944 9158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:28.722879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.65:36005 (local address 127.3.243.67:36017)
0619 18:47:28.723044 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:28.723063 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:28.723154 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:28.723506 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:28.723511 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:28.723543 (+ 32us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:47:28.724211 (+ 668us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:47:28.724214 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:47:28.724231 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:47:28.724278 (+ 47us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:47:28.724553 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:47:28.724556 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:47:28.724559 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 18:47:28.724637 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 18:47:28.724678 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260619 18:47:28.724969 9170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:28.722988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.67:36017 (local address 127.3.243.65:36005)
0619 18:47:28.723147 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:28.723153 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:28.723177 (+ 24us) server_negotiation.cc:407] Connection header received
0619 18:47:28.723279 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:28.723284 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:28.723350 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:28.723456 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:28.724411 (+ 955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:28.724415 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:28.724418 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:47:28.724433 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:28.724464 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:47:28.724476 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:28.724572 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:28.724650 (+ 78us) server_negotiation.cc:299] Negotiation successful
0619 18:47:28.724709 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 18:47:28.725944 9180 raft_consensus.cc:482] T 3df4b1fce6b64155b135ebee03bb104f P 1681097497944bd6a6b64c9620d806eb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:47:28.726482 9172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:28.723408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.67:35651 (local address 127.3.243.66:37997)
0619 18:47:28.723582 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:28.723588 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:28.724063 (+ 475us) server_negotiation.cc:407] Connection header received
0619 18:47:28.724217 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:28.724222 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:28.724279 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:28.724373 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:28.725882 (+ 1509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:28.725888 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:28.725892 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:47:28.725908 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:28.725939 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:47:28.725950 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:28.726054 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:28.726142 (+ 88us) server_negotiation.cc:299] Negotiation successful
0619 18:47:28.726195 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 18:47:28.726325 8851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3df4b1fce6b64155b135ebee03bb104f" candidate_uuid: "2468bf4f7fd7442c9020a9624c3d5fff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06baba1b3eb44ba3b71533629f6d5c9f" is_pre_election: true
I20260619 18:47:28.726595 9182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:28.723221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.66:37997 (local address 127.3.243.67:35651)
0619 18:47:28.723967 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:28.723984 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:28.724085 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:28.724650 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:28.724654 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:28.724678 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:47:28.725651 (+ 973us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:47:28.725655 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:47:28.725670 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:47:28.725711 (+ 41us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:47:28.726030 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:47:28.726034 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:47:28.726038 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 18:47:28.726133 (+ 95us) client_negotiation.cc:240] Negotiation successful
0619 18:47:28.726179 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"thread_start_us":461,"threads_started":1}
I20260619 18:47:28.726332 9180 raft_consensus.cc:504] T 3df4b1fce6b64155b135ebee03bb104f P 1681097497944bd6a6b64c9620d806eb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2468bf4f7fd7442c9020a9624c3d5fff" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 43533 } } peers { permanent_uuid: "06baba1b3eb44ba3b71533629f6d5c9f" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 36005 } } peers { permanent_uuid: "1681097497944bd6a6b64c9620d806eb" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 37997 } }
I20260619 18:47:28.726838 8851 raft_consensus.cc:2454] T 3df4b1fce6b64155b135ebee03bb104f P 06baba1b3eb44ba3b71533629f6d5c9f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2468bf4f7fd7442c9020a9624c3d5fff in term 0.
I20260619 18:47:28.727466 9047 leader_election.cc:302] T 3df4b1fce6b64155b135ebee03bb104f P 2468bf4f7fd7442c9020a9624c3d5fff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06baba1b3eb44ba3b71533629f6d5c9f, 2468bf4f7fd7442c9020a9624c3d5fff; no voters:
I20260619 18:47:28.727813 9179 raft_consensus.cc:2781] T 3df4b1fce6b64155b135ebee03bb104f P 2468bf4f7fd7442c9020a9624c3d5fff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:47:28.727948 9179 raft_consensus.cc:482] T 3df4b1fce6b64155b135ebee03bb104f P 2468bf4f7fd7442c9020a9624c3d5fff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:47:28.728040 9179 raft_consensus.cc:3037] T 3df4b1fce6b64155b135ebee03bb104f P 2468bf4f7fd7442c9020a9624c3d5fff [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:47:28.727969 8981 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3df4b1fce6b64155b135ebee03bb104f" candidate_uuid: "2468bf4f7fd7442c9020a9624c3d5fff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1681097497944bd6a6b64c9620d806eb" is_pre_election: true
I20260619 18:47:28.728425 9180 leader_election.cc:288] T 3df4b1fce6b64155b135ebee03bb104f P 1681097497944bd6a6b64c9620d806eb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2468bf4f7fd7442c9020a9624c3d5fff (127.3.243.67:43533), 06baba1b3eb44ba3b71533629f6d5c9f (127.3.243.65:36005)
I20260619 18:47:28.728420 8981 raft_consensus.cc:2454] T 3df4b1fce6b64155b135ebee03bb104f P 1681097497944bd6a6b64c9620d806eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2468bf4f7fd7442c9020a9624c3d5fff in term 0.
I20260619 18:47:28.730858 9171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:28.729094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.66:50623 (local address 127.3.243.67:43533)
0619 18:47:28.729256 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:28.729261 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:28.729392 (+ 131us) server_negotiation.cc:407] Connection header received
0619 18:47:28.729477 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:28.729481 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:28.729538 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:28.729642 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:28.730368 (+ 726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:28.730373 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:28.730378 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:47:28.730395 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:28.730430 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:47:28.730444 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:28.730558 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:28.730593 (+ 35us) server_negotiation.cc:299] Negotiation successful
0619 18:47:28.730638 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 18:47:28.730916 9183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:28.728680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.67:43533 (local address 127.3.243.66:50623)
0619 18:47:28.729306 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:28.729320 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:28.729403 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:28.729647 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:28.729649 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:28.729666 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:47:28.730202 (+ 536us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:47:28.730204 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:47:28.730215 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:47:28.730247 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:47:28.730524 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:47:28.730527 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:47:28.730529 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 18:47:28.730585 (+ 56us) client_negotiation.cc:240] Negotiation successful
0619 18:47:28.730620 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":358,"threads_started":1}
I20260619 18:47:28.731072 9179 raft_consensus.cc:504] T 3df4b1fce6b64155b135ebee03bb104f P 2468bf4f7fd7442c9020a9624c3d5fff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2468bf4f7fd7442c9020a9624c3d5fff" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 43533 } } peers { permanent_uuid: "06baba1b3eb44ba3b71533629f6d5c9f" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 36005 } } peers { permanent_uuid: "1681097497944bd6a6b64c9620d806eb" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 37997 } }
I20260619 18:47:28.731727 9179 leader_election.cc:288] T 3df4b1fce6b64155b135ebee03bb104f P 2468bf4f7fd7442c9020a9624c3d5fff [CANDIDATE]: Term 1 election: Requested vote from peers 06baba1b3eb44ba3b71533629f6d5c9f (127.3.243.65:36005), 1681097497944bd6a6b64c9620d806eb (127.3.243.66:37997)
I20260619 18:47:28.731684 9111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3df4b1fce6b64155b135ebee03bb104f" candidate_uuid: "1681097497944bd6a6b64c9620d806eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2468bf4f7fd7442c9020a9624c3d5fff" is_pre_election: true
I20260619 18:47:28.732175 9111 raft_consensus.cc:2379] T 3df4b1fce6b64155b135ebee03bb104f P 2468bf4f7fd7442c9020a9624c3d5fff [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1681097497944bd6a6b64c9620d806eb in current term 1: Already voted for candidate 2468bf4f7fd7442c9020a9624c3d5fff in this term.
I20260619 18:47:28.732199 8851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3df4b1fce6b64155b135ebee03bb104f" candidate_uuid: "2468bf4f7fd7442c9020a9624c3d5fff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06baba1b3eb44ba3b71533629f6d5c9f"
I20260619 18:47:28.732410 8851 raft_consensus.cc:3037] T 3df4b1fce6b64155b135ebee03bb104f P 06baba1b3eb44ba3b71533629f6d5c9f [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:47:28.732721 8981 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3df4b1fce6b64155b135ebee03bb104f" candidate_uuid: "2468bf4f7fd7442c9020a9624c3d5fff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1681097497944bd6a6b64c9620d806eb"
I20260619 18:47:28.732908 8981 raft_consensus.cc:3037] T 3df4b1fce6b64155b135ebee03bb104f P 1681097497944bd6a6b64c9620d806eb [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:47:28.735234 9183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:28.732191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.65:36005 (local address 127.3.243.66:41845)
0619 18:47:28.733142 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:28.733161 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:28.733308 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:28.733812 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:28.733817 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:28.733865 (+ 48us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:47:28.734431 (+ 566us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:47:28.734435 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:47:28.734452 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:47:28.734504 (+ 52us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:47:28.734841 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:47:28.734846 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:47:28.734851 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 18:47:28.734944 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 18:47:28.734993 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":833}
I20260619 18:47:28.735275 9170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:28.732196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.66:41845 (local address 127.3.243.65:36005)
0619 18:47:28.732357 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:28.732362 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:28.733225 (+ 863us) server_negotiation.cc:407] Connection header received
0619 18:47:28.733445 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:28.733451 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:28.733525 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:28.733589 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:28.734656 (+ 1067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:28.734662 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:28.734666 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:47:28.734684 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:28.734721 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:47:28.734735 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:28.734858 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:28.734949 (+ 91us) server_negotiation.cc:299] Negotiation successful
0619 18:47:28.735002 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 18:47:28.735586 8851 raft_consensus.cc:2454] T 3df4b1fce6b64155b135ebee03bb104f P 06baba1b3eb44ba3b71533629f6d5c9f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2468bf4f7fd7442c9020a9624c3d5fff in term 1.
I20260619 18:47:28.735792 8981 raft_consensus.cc:2454] T 3df4b1fce6b64155b135ebee03bb104f P 1681097497944bd6a6b64c9620d806eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2468bf4f7fd7442c9020a9624c3d5fff in term 1.
I20260619 18:47:28.735952 9047 leader_election.cc:302] T 3df4b1fce6b64155b135ebee03bb104f P 2468bf4f7fd7442c9020a9624c3d5fff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06baba1b3eb44ba3b71533629f6d5c9f, 2468bf4f7fd7442c9020a9624c3d5fff; no voters:
I20260619 18:47:28.736299 9179 raft_consensus.cc:2781] T 3df4b1fce6b64155b135ebee03bb104f P 2468bf4f7fd7442c9020a9624c3d5fff [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:47:28.736442 8851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3df4b1fce6b64155b135ebee03bb104f" candidate_uuid: "1681097497944bd6a6b64c9620d806eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06baba1b3eb44ba3b71533629f6d5c9f" is_pre_election: true
I20260619 18:47:28.736738 8851 raft_consensus.cc:2379] T 3df4b1fce6b64155b135ebee03bb104f P 06baba1b3eb44ba3b71533629f6d5c9f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1681097497944bd6a6b64c9620d806eb in current term 1: Already voted for candidate 2468bf4f7fd7442c9020a9624c3d5fff in this term.
I20260619 18:47:28.737445 8917 leader_election.cc:302] T 3df4b1fce6b64155b135ebee03bb104f P 1681097497944bd6a6b64c9620d806eb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1681097497944bd6a6b64c9620d806eb; no voters: 06baba1b3eb44ba3b71533629f6d5c9f, 2468bf4f7fd7442c9020a9624c3d5fff
I20260619 18:47:28.738029 9180 raft_consensus.cc:2726] T 3df4b1fce6b64155b135ebee03bb104f P 1681097497944bd6a6b64c9620d806eb [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260619 18:47:28.740415 9179 raft_consensus.cc:686] T 3df4b1fce6b64155b135ebee03bb104f P 2468bf4f7fd7442c9020a9624c3d5fff [term 1 LEADER]: Becoming Leader. State: Replica: 2468bf4f7fd7442c9020a9624c3d5fff, State: Running, Role: LEADER
I20260619 18:47:28.740963 9179 consensus_queue.cc:237] T 3df4b1fce6b64155b135ebee03bb104f P 2468bf4f7fd7442c9020a9624c3d5fff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2468bf4f7fd7442c9020a9624c3d5fff" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 43533 } } peers { permanent_uuid: "06baba1b3eb44ba3b71533629f6d5c9f" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 36005 } } peers { permanent_uuid: "1681097497944bd6a6b64c9620d806eb" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 37997 } }
I20260619 18:47:28.745532 8709 catalog_manager.cc:5495] T 3df4b1fce6b64155b135ebee03bb104f P 2468bf4f7fd7442c9020a9624c3d5fff reported cstate change: term changed from 0 to 1, leader changed from <none> to 2468bf4f7fd7442c9020a9624c3d5fff (127.3.243.67). New cstate: current_term: 1 leader_uuid: "2468bf4f7fd7442c9020a9624c3d5fff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2468bf4f7fd7442c9020a9624c3d5fff" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 43533 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06baba1b3eb44ba3b71533629f6d5c9f" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 36005 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1681097497944bd6a6b64c9620d806eb" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 37997 } health_report { overall_health: UNKNOWN } } }
I20260619 18:47:28.765269 9171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:28.763175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39790 (local address 127.3.243.67:43533)
0619 18:47:28.763366 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:28.763372 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:28.763469 (+ 97us) server_negotiation.cc:407] Connection header received
0619 18:47:28.763672 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:28.763676 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:28.763769 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:28.763858 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:28.764667 (+ 809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:28.764671 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:28.764674 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:47:28.764688 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:28.764719 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:47:28.764731 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:28.764842 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:28.764947 (+ 105us) server_negotiation.cc:299] Negotiation successful
0619 18:47:28.765002 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 18:47:28.772867 8851 raft_consensus.cc:1261] T 3df4b1fce6b64155b135ebee03bb104f P 06baba1b3eb44ba3b71533629f6d5c9f [term 1 FOLLOWER]: Refusing update from remote peer 2468bf4f7fd7442c9020a9624c3d5fff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:47:28.772902 8981 raft_consensus.cc:1261] T 3df4b1fce6b64155b135ebee03bb104f P 1681097497944bd6a6b64c9620d806eb [term 1 FOLLOWER]: Refusing update from remote peer 2468bf4f7fd7442c9020a9624c3d5fff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:47:28.773629 9184 consensus_queue.cc:1035] T 3df4b1fce6b64155b135ebee03bb104f P 2468bf4f7fd7442c9020a9624c3d5fff [LEADER]: Connected to new peer: Peer: permanent_uuid: "06baba1b3eb44ba3b71533629f6d5c9f" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 36005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:47:28.773998 9179 consensus_queue.cc:1035] T 3df4b1fce6b64155b135ebee03bb104f P 2468bf4f7fd7442c9020a9624c3d5fff [LEADER]: Connected to new peer: Peer: permanent_uuid: "1681097497944bd6a6b64c9620d806eb" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 37997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 18:47:28.778487 8897 tablet.cc:2369] T 3df4b1fce6b64155b135ebee03bb104f P 06baba1b3eb44ba3b71533629f6d5c9f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:47:28.802877 8709 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40508:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:47:28.803122 8709 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40508:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:47:28.806669 8709 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d1f4ff5396fc478faef57efb46462e51: Sending DeleteTablet for 3 replicas of tablet 3df4b1fce6b64155b135ebee03bb104f
I20260619 18:47:28.807706 9091 tablet_service.cc:1514] Processing DeleteTablet for tablet 3df4b1fce6b64155b135ebee03bb104f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:47:28 UTC) from {username='slave'} at 127.0.0.1:39786
I20260619 18:47:28.807824 8831 tablet_service.cc:1514] Processing DeleteTablet for tablet 3df4b1fce6b64155b135ebee03bb104f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:47:28 UTC) from {username='slave'} at 127.0.0.1:40372
I20260619 18:47:28.808472 8961 tablet_service.cc:1514] Processing DeleteTablet for tablet 3df4b1fce6b64155b135ebee03bb104f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:47:28 UTC) from {username='slave'} at 127.0.0.1:50824
I20260619 18:47:28.808746 9196 tablet_replica.cc:330] T 3df4b1fce6b64155b135ebee03bb104f P 2468bf4f7fd7442c9020a9624c3d5fff: stopping tablet replica
I20260619 18:47:28.809211 9196 raft_consensus.cc:2229] T 3df4b1fce6b64155b135ebee03bb104f P 2468bf4f7fd7442c9020a9624c3d5fff [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:47:28.809887 9196 raft_consensus.cc:2258] T 3df4b1fce6b64155b135ebee03bb104f P 2468bf4f7fd7442c9020a9624c3d5fff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:47:28.810590 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 8769
I20260619 18:47:28.815666 9196 ts_tablet_manager.cc:1905] T 3df4b1fce6b64155b135ebee03bb104f P 2468bf4f7fd7442c9020a9624c3d5fff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:47:28.827247 9198 tablet_replica.cc:330] T 3df4b1fce6b64155b135ebee03bb104f P 1681097497944bd6a6b64c9620d806eb: stopping tablet replica
I20260619 18:47:28.827744 9198 raft_consensus.cc:2229] T 3df4b1fce6b64155b135ebee03bb104f P 1681097497944bd6a6b64c9620d806eb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:47:28.828316 9198 raft_consensus.cc:2258] T 3df4b1fce6b64155b135ebee03bb104f P 1681097497944bd6a6b64c9620d806eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:47:28.829522 9196 ts_tablet_manager.cc:1918] T 3df4b1fce6b64155b135ebee03bb104f P 2468bf4f7fd7442c9020a9624c3d5fff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:47:28.829715 9196 log.cc:1196] T 3df4b1fce6b64155b135ebee03bb104f P 2468bf4f7fd7442c9020a9624c3d5fff: Deleting WAL directory at /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894810715282-4045-0/minicluster-data/ts-2/wal/wals/3df4b1fce6b64155b135ebee03bb104f
I20260619 18:47:28.830252 9196 ts_tablet_manager.cc:1939] T 3df4b1fce6b64155b135ebee03bb104f P 2468bf4f7fd7442c9020a9624c3d5fff: Deleting consensus metadata
I20260619 18:47:28.830854 9170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:28.811811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40382 (local address 127.3.243.65:36005)
0619 18:47:28.811980 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:28.811986 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:28.827523 (+ 15537us) server_negotiation.cc:407] Connection header received
0619 18:47:28.828242 (+ 719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:28.828249 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:28.828352 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:28.828456 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:28.829964 (+ 1508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:28.829971 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:28.829976 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:47:28.829998 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:28.830038 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:47:28.830054 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:28.830157 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:28.830497 (+ 340us) server_negotiation.cc:299] Negotiation successful
0619 18:47:28.830569 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 18:47:28.832091 8891 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:47:28.832093 8696 catalog_manager.cc:4841] TS 2468bf4f7fd7442c9020a9624c3d5fff (127.3.243.67:43533): tablet 3df4b1fce6b64155b135ebee03bb104f (table test-table [id=e021131c5ea0460a90d8d5a42b101875]) successfully deleted
I20260619 18:47:28.832569 9198 ts_tablet_manager.cc:1905] T 3df4b1fce6b64155b135ebee03bb104f P 1681097497944bd6a6b64c9620d806eb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:47:28.840458 9198 ts_tablet_manager.cc:1918] T 3df4b1fce6b64155b135ebee03bb104f P 1681097497944bd6a6b64c9620d806eb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:47:28.840626 9198 log.cc:1196] T 3df4b1fce6b64155b135ebee03bb104f P 1681097497944bd6a6b64c9620d806eb: Deleting WAL directory at /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894810715282-4045-0/minicluster-data/ts-1/wal/wals/3df4b1fce6b64155b135ebee03bb104f
I20260619 18:47:28.841116 9198 ts_tablet_manager.cc:1939] T 3df4b1fce6b64155b135ebee03bb104f P 1681097497944bd6a6b64c9620d806eb: Deleting consensus metadata
I20260619 18:47:28.843003 8697 catalog_manager.cc:4841] TS 1681097497944bd6a6b64c9620d806eb (127.3.243.66:37997): tablet 3df4b1fce6b64155b135ebee03bb104f (table test-table [id=e021131c5ea0460a90d8d5a42b101875]) successfully deleted
I20260619 18:47:28.921993 9197 tablet_replica.cc:330] T 3df4b1fce6b64155b135ebee03bb104f P 06baba1b3eb44ba3b71533629f6d5c9f: stopping tablet replica
I20260619 18:47:28.922354 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 8769
W20260619 18:47:28.936255 8697 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.243.65:36005 (error 108)
I20260619 18:47:28.937053 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 8899
I20260619 18:47:28.938494 9169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:28.937781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.65:36005 (local address 127.0.0.1:40394)
0619 18:47:28.938068 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:28.938232 (+ 164us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.65:36005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148}
W20260619 18:47:28.938774 8697 catalog_manager.cc:4584] TS 06baba1b3eb44ba3b71533629f6d5c9f (127.3.243.65:36005): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3df4b1fce6b64155b135ebee03bb104f: Network error: Client connection negotiation failed: client connection to 127.3.243.65:36005: connect: Connection refused (error 111)
I20260619 18:47:28.940286 9172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:28.937903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50830 (local address 127.3.243.66:37997)
0619 18:47:28.938074 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:28.938080 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:28.938232 (+ 152us) server_negotiation.cc:407] Connection header received
0619 18:47:28.938436 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:28.938441 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:28.938554 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:28.938673 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:28.939642 (+ 969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:28.939647 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:28.939650 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:47:28.939667 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:28.939701 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:47:28.939713 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:28.939827 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:28.939927 (+ 100us) server_negotiation.cc:299] Negotiation successful
0619 18:47:28.939973 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 18:47:28.940936 9021 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:47:28.992697 9169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:28.992114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.65:36005 (local address 127.0.0.1:40396)
0619 18:47:28.992405 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:28.992529 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.65:36005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
I20260619 18:47:28.993920 9169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:28.993489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.65:36005 (local address 127.0.0.1:40412)
0619 18:47:28.993653 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:28.993742 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.65:36005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260619 18:47:29.030195 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 8899
I20260619 18:47:29.044364 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 9029
I20260619 18:47:29.047286 9171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:29.045336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39792 (local address 127.3.243.67:43533)
0619 18:47:29.045492 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:29.045496 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:29.045513 (+ 17us) server_negotiation.cc:407] Connection header received
0619 18:47:29.045631 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:29.045635 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:29.045683 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:29.045776 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:29.046700 (+ 924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:29.046705 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:29.046708 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:47:29.046722 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:29.046755 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:47:29.046770 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:29.046866 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:29.046962 (+ 96us) server_negotiation.cc:299] Negotiation successful
0619 18:47:29.047007 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 18:47:29.047973 9151 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:47:29.054383 9169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:29.053901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.65:36005 (local address 127.0.0.1:40416)
0619 18:47:29.054102 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:29.054206 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.65:36005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260619 18:47:29.055434 9169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:29.055089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.65:36005 (local address 127.0.0.1:40424)
0619 18:47:29.055229 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:29.055304 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.65:36005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260619 18:47:29.135910 9169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:29.135398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.65:36005 (local address 127.0.0.1:40426)
0619 18:47:29.135646 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:29.135754 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.65:36005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260619 18:47:29.137022 9169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:29.136579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.65:36005 (local address 127.0.0.1:40430)
0619 18:47:29.136794 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:29.136886 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.65:36005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
I20260619 18:47:29.165398 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 9029
I20260619 18:47:29.180096 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 8679
I20260619 18:47:29.181228 8740 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:47:29.286971 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 8679
2026-06-19T18:47:29Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2454 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260619 18:47:29.323498 4045 mini_oidc.cc:152] Starting JWKS server
I20260619 18:47:29.326025 4045 webserver.cc:460] Webserver started at https://127.0.0.1:38861/ using document root <none> and password file <none>
I20260619 18:47:29.326305 4045 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 18:47:29.326891 4045 webserver.cc:460] Webserver started at http://127.0.0.1:37531/ using document root <none> and password file <none>
2026-06-19T18:47:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:47:29Z Disabled control of system clock
I20260619 18:47:29.350865 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781894810715282-4045-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781894810715282-4045-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781894810715282-4045-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.126:36431
--webserver_interface=127.3.243.126
--webserver_port=0
--builtin_ntp_servers=127.3.243.84:43087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.126:36431
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38861/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781894810715282-4045-0/testchainca.cert with env {}
W20260619 18:47:29.658679 9210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:47:29.658967 9210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:47:29.659051 9210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:47:29.666296 9210 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:47:29.666399 9210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:47:29.666445 9210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:47:29.666498 9210 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:47:29.666539 9210 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 18:47:29.666595 9210 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 18:47:29.666632 9210 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38861/jwks.json
I20260619 18:47:29.676620 9210 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:43087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781894810715282-4045-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781894810715282-4045-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.243.126:36431
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.126:36431
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38861/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.243.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781894810715282-4045-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781894810715282-4045-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:47:29.677860 9210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:47:29.679149 9210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:47:29.690155 9215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:29.690069 9218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:29.690105 9216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:47:29.690901 9210 server_base.cc:1034] running on GCE node
I20260619 18:47:29.691630 9210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:47:29.692940 9210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:47:29.694175 9210 hybrid_clock.cc:648] HybridClock initialized: now 1781894849694068 us; error 60 us; skew 500 ppm
I20260619 18:47:29.694499 9210 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 18:47:29.696677 9210 webserver.cc:460] Webserver started at http://127.3.243.126:37029/ using document root <none> and password file <none>
I20260619 18:47:29.697311 9210 fs_manager.cc:359] Metadata directory not provided
I20260619 18:47:29.697402 9210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:47:29.697682 9210 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:47:29.700488 9210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781894810715282-4045-0/minicluster-data/master-0/data/instance:
uuid: "c2d34b085c8f4312a8e848e1fea0275f"
format_stamp: "Formatted at 2026-06-19 18:47:29 on dist-test-slave-m9wg"
server_key: "280777e49132321fd866500f350f716e"
server_key_iv: "14194538dce68abc4de9e46b72cb12f1"
server_key_version: "encryptionkey@0"
I20260619 18:47:29.701205 9210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781894810715282-4045-0/minicluster-data/master-0/wal/instance:
uuid: "c2d34b085c8f4312a8e848e1fea0275f"
format_stamp: "Formatted at 2026-06-19 18:47:29 on dist-test-slave-m9wg"
server_key: "280777e49132321fd866500f350f716e"
server_key_iv: "14194538dce68abc4de9e46b72cb12f1"
server_key_version: "encryptionkey@0"
I20260619 18:47:29.705806 9210 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 18:47:29.709023 9224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:29.710258 9210 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 18:47:29.710426 9210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781894810715282-4045-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781894810715282-4045-0/minicluster-data/master-0/wal
uuid: "c2d34b085c8f4312a8e848e1fea0275f"
format_stamp: "Formatted at 2026-06-19 18:47:29 on dist-test-slave-m9wg"
server_key: "280777e49132321fd866500f350f716e"
server_key_iv: "14194538dce68abc4de9e46b72cb12f1"
server_key_version: "encryptionkey@0"
I20260619 18:47:29.710615 9210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781894810715282-4045-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781894810715282-4045-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781894810715282-4045-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:47:29.732738 9210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:47:29.733536 9210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:47:29.733817 9210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:47:29.752516 9210 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.126:36431
I20260619 18:47:29.752537 9275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.126:36431 every 8 connection(s)
I20260619 18:47:29.753748 9210 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
I20260619 18:47:29.756850 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 9210
I20260619 18:47:29.757042 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781894810715282-4045-0/minicluster-data/master-0/wal/instance
I20260619 18:47:29.757400 9276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:47:29.757645 4045 external_mini_cluster.cc:1428] Setting key 022d5dcebb181835f24c7a251f255b44
I20260619 18:47:29.771297 9276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2d34b085c8f4312a8e848e1fea0275f: Bootstrap starting.
I20260619 18:47:29.771543 9279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:29.759388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34028 (local address 127.3.243.126:36431)
0619 18:47:29.760382 (+ 994us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:29.760392 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:29.760437 (+ 45us) server_negotiation.cc:407] Connection header received
0619 18:47:29.761998 (+ 1561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:29.762023 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:29.762607 (+ 584us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:29.763036 (+ 429us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:29.763563 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:29.765506 (+ 1943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:29.766793 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:29.767771 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:29.767956 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:29.768964 (+ 1008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:29.768974 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:29.768992 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:47:29.769652 (+ 660us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:29.769746 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:47:29.769771 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:29.769890 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:29.770029 (+ 139us) server_negotiation.cc:299] Negotiation successful
0619 18:47:29.770323 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":325,"threads_started":1}
I20260619 18:47:29.774895 9276 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c2d34b085c8f4312a8e848e1fea0275f: Neither blocks nor log segments found. Creating new log.
I20260619 18:47:29.775341 4045 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:47:29.775913 9276 log.cc:824] T 00000000000000000000000000000000 P c2d34b085c8f4312a8e848e1fea0275f: Log is configured to *not* fsync() on all Append() calls
I20260619 18:47:29.778829 9276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2d34b085c8f4312a8e848e1fea0275f: No bootstrap required, opened a new log
I20260619 18:47:29.785962 9276 raft_consensus.cc:348] T 00000000000000000000000000000000 P c2d34b085c8f4312a8e848e1fea0275f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2d34b085c8f4312a8e848e1fea0275f" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 36431 } }
I20260619 18:47:29.786311 9276 raft_consensus.cc:374] T 00000000000000000000000000000000 P c2d34b085c8f4312a8e848e1fea0275f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:47:29.786406 9276 raft_consensus.cc:729] T 00000000000000000000000000000000 P c2d34b085c8f4312a8e848e1fea0275f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2d34b085c8f4312a8e848e1fea0275f, State: Initialized, Role: FOLLOWER
I20260619 18:47:29.786933 9276 consensus_queue.cc:260] T 00000000000000000000000000000000 P c2d34b085c8f4312a8e848e1fea0275f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2d34b085c8f4312a8e848e1fea0275f" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 36431 } }
I20260619 18:47:29.787129 9276 raft_consensus.cc:388] T 00000000000000000000000000000000 P c2d34b085c8f4312a8e848e1fea0275f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:47:29.787252 9276 raft_consensus.cc:482] T 00000000000000000000000000000000 P c2d34b085c8f4312a8e848e1fea0275f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:47:29.787405 9276 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c2d34b085c8f4312a8e848e1fea0275f [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:47:29.789569 9276 raft_consensus.cc:504] T 00000000000000000000000000000000 P c2d34b085c8f4312a8e848e1fea0275f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2d34b085c8f4312a8e848e1fea0275f" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 36431 } }
I20260619 18:47:29.790030 9276 leader_election.cc:302] T 00000000000000000000000000000000 P c2d34b085c8f4312a8e848e1fea0275f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c2d34b085c8f4312a8e848e1fea0275f; no voters:
I20260619 18:47:29.790587 9276 leader_election.cc:288] T 00000000000000000000000000000000 P c2d34b085c8f4312a8e848e1fea0275f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:47:29.790726 9281 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c2d34b085c8f4312a8e848e1fea0275f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:47:29.791436 9281 raft_consensus.cc:686] T 00000000000000000000000000000000 P c2d34b085c8f4312a8e848e1fea0275f [term 1 LEADER]: Becoming Leader. State: Replica: c2d34b085c8f4312a8e848e1fea0275f, State: Running, Role: LEADER
I20260619 18:47:29.791931 9281 consensus_queue.cc:237] T 00000000000000000000000000000000 P c2d34b085c8f4312a8e848e1fea0275f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2d34b085c8f4312a8e848e1fea0275f" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 36431 } }
I20260619 18:47:29.792503 9276 sys_catalog.cc:564] T 00000000000000000000000000000000 P c2d34b085c8f4312a8e848e1fea0275f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:47:29.794565 9283 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2d34b085c8f4312a8e848e1fea0275f [sys.catalog]: SysCatalogTable state changed. Reason: New leader c2d34b085c8f4312a8e848e1fea0275f. Latest consensus state: current_term: 1 leader_uuid: "c2d34b085c8f4312a8e848e1fea0275f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2d34b085c8f4312a8e848e1fea0275f" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 36431 } } }
I20260619 18:47:29.795077 9283 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2d34b085c8f4312a8e848e1fea0275f [sys.catalog]: This master's current role is: LEADER
I20260619 18:47:29.794935 9282 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2d34b085c8f4312a8e848e1fea0275f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c2d34b085c8f4312a8e848e1fea0275f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2d34b085c8f4312a8e848e1fea0275f" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 36431 } } }
I20260619 18:47:29.795190 9282 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2d34b085c8f4312a8e848e1fea0275f [sys.catalog]: This master's current role is: LEADER
I20260619 18:47:29.797020 9290 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:47:29.802093 9290 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:47:29.810817 9290 catalog_manager.cc:1347] Generated new cluster ID: 50424c468f344873965601db6ecd97c0
I20260619 18:47:29.810940 9290 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:47:29.820451 9290 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:47:29.821651 9290 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:47:29.833658 9290 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c2d34b085c8f4312a8e848e1fea0275f: Generated new TSK 0
I20260619 18:47:29.834617 9290 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 18:47:29.877535 9306 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260619 18:47:29.877717 9306 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 18:47:29.878729 9279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:29.859091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48292 (local address 127.3.243.126:36431)
0619 18:47:29.859241 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:29.859246 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:29.859754 (+ 508us) server_negotiation.cc:407] Connection header received
0619 18:47:29.860064 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:29.860068 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:29.860138 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:29.860290 (+ 152us) server_negotiation.cc:226] Negotiated authn=JWT
0619 18:47:29.860786 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:29.861493 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:29.862841 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:29.863266 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:29.863370 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:29.863853 (+ 483us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 18:47:29.878074 (+ 14221us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38861/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 18:47:29.878403 (+ 329us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48292: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38861/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":47}
W20260619 18:47:29.878870 9279 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48292: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38861/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260619 18:47:29.880259 9305 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:47:29.859079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:36431 (local address 127.0.0.1:48292)
0619 18:47:29.859644 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:29.859663 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:29.859829 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:29.860330 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:29.860335 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:29.860347 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0619 18:47:29.860637 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:29.860645 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:29.861627 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:29.861631 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:29.862698 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:29.862708 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:29.863412 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:29.863419 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:29.863553 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:29.863607 (+ 54us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 18:47:29.878370 (+ 14763us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38861/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 18:47:29.878703 (+ 333us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.243.126:36431: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38861/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":292,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.3.243.126:36431: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38861/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260619 18:47:29.881973 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 9210
I20260619 18:47:29.882944 9271 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:47:29.974648 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 9210
2026-06-19T18:47:30Z chronyd exiting
I20260619 18:47:30.021046 4045 test_util.cc:175] -----------------------------------------------
I20260619 18:47:30.021183 4045 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781894810715282-4045-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (701 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260619 18:47:30.024685 4045 mini_oidc.cc:152] Starting JWKS server
I20260619 18:47:30.026428 4045 webserver.cc:460] Webserver started at https://127.0.0.1:36923/ using document root <none> and password file <none>
I20260619 18:47:30.026623 4045 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 18:47:30.027310 4045 webserver.cc:460] Webserver started at http://127.0.0.1:38009/ using document root <none> and password file <none>
2026-06-19T18:47:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:47:30Z Disabled control of system clock
I20260619 18:47:30.051594 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781894810715282-4045-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781894810715282-4045-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781894810715282-4045-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.126:40903
--webserver_interface=127.3.243.126
--webserver_port=0
--builtin_ntp_servers=127.3.243.84:43543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.126:40903
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36923/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781894810715282-4045-0/testchainca.cert with env {}
W20260619 18:47:30.353096 9313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:47:30.353421 9313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:47:30.353559 9313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:47:30.360708 9313 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:47:30.360814 9313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:47:30.360877 9313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:47:30.360952 9313 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:47:30.361011 9313 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 18:47:30.361092 9313 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 18:47:30.361142 9313 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36923/jwks.json
I20260619 18:47:30.370855 9313 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:43543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781894810715282-4045-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781894810715282-4045-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.243.126:40903
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.126:40903
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36923/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.243.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781894810715282-4045-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781894810715282-4045-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:47:30.372102 9313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:47:30.373413 9313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:47:30.384342 9318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:30.384392 9321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:30.384446 9319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:47:30.385169 9313 server_base.cc:1034] running on GCE node
I20260619 18:47:30.385845 9313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:47:30.387109 9313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:47:30.388290 9313 hybrid_clock.cc:648] HybridClock initialized: now 1781894850388241 us; error 38 us; skew 500 ppm
I20260619 18:47:30.388608 9313 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 18:47:30.390703 9313 webserver.cc:460] Webserver started at http://127.3.243.126:33189/ using document root <none> and password file <none>
I20260619 18:47:30.391333 9313 fs_manager.cc:359] Metadata directory not provided
I20260619 18:47:30.391434 9313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:47:30.391736 9313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:47:30.394507 9313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781894810715282-4045-0/minicluster-data/master-0/data/instance:
uuid: "cd505214823c4932998fdead415c475f"
format_stamp: "Formatted at 2026-06-19 18:47:30 on dist-test-slave-m9wg"
server_key: "f3197b0a3a47ea3380176ecc1e97a914"
server_key_iv: "8f73858dfc6338c9bd84bfe773e4d1d0"
server_key_version: "encryptionkey@0"
I20260619 18:47:30.395260 9313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781894810715282-4045-0/minicluster-data/master-0/wal/instance:
uuid: "cd505214823c4932998fdead415c475f"
format_stamp: "Formatted at 2026-06-19 18:47:30 on dist-test-slave-m9wg"
server_key: "f3197b0a3a47ea3380176ecc1e97a914"
server_key_iv: "8f73858dfc6338c9bd84bfe773e4d1d0"
server_key_version: "encryptionkey@0"
I20260619 18:47:30.400171 9313 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 18:47:30.403496 9327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:30.404983 9313 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:47:30.405145 9313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781894810715282-4045-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781894810715282-4045-0/minicluster-data/master-0/wal
uuid: "cd505214823c4932998fdead415c475f"
format_stamp: "Formatted at 2026-06-19 18:47:30 on dist-test-slave-m9wg"
server_key: "f3197b0a3a47ea3380176ecc1e97a914"
server_key_iv: "8f73858dfc6338c9bd84bfe773e4d1d0"
server_key_version: "encryptionkey@0"
I20260619 18:47:30.405349 9313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781894810715282-4045-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781894810715282-4045-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781894810715282-4045-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:47:30.422251 9313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:47:30.423071 9313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:47:30.423352 9313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:47:30.440076 9313 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.126:40903
I20260619 18:47:30.440097 9378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.126:40903 every 8 connection(s)
I20260619 18:47:30.441425 9313 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
I20260619 18:47:30.444586 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 9313
I20260619 18:47:30.444782 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781894810715282-4045-0/minicluster-data/master-0/wal/instance
I20260619 18:47:30.445185 9379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:47:30.445465 4045 external_mini_cluster.cc:1428] Setting key d9335120106dc019aa3d44e634bd833e
I20260619 18:47:30.455451 9379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd505214823c4932998fdead415c475f: Bootstrap starting.
I20260619 18:47:30.458778 9379 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd505214823c4932998fdead415c475f: Neither blocks nor log segments found. Creating new log.
I20260619 18:47:30.459993 9379 log.cc:824] T 00000000000000000000000000000000 P cd505214823c4932998fdead415c475f: Log is configured to *not* fsync() on all Append() calls
I20260619 18:47:30.462818 9379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd505214823c4932998fdead415c475f: No bootstrap required, opened a new log
I20260619 18:47:30.467145 9382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:30.447454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41266 (local address 127.3.243.126:40903)
0619 18:47:30.448720 (+ 1266us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:30.448735 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:30.456236 (+ 7501us) server_negotiation.cc:407] Connection header received
0619 18:47:30.458088 (+ 1852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:30.458110 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:30.458582 (+ 472us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:30.458951 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:30.459498 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:30.460875 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:30.462176 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:30.463171 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:30.463359 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:30.464332 (+ 973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:30.464342 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:30.464364 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:47:30.465116 (+ 752us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:30.465211 (+ 95us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:47:30.465236 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:30.465353 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:30.465545 (+ 192us) server_negotiation.cc:299] Negotiation successful
0619 18:47:30.465895 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":722,"thread_start_us":437,"threads_started":1}
I20260619 18:47:30.470901 4045 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:47:30.471912 9379 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd505214823c4932998fdead415c475f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd505214823c4932998fdead415c475f" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 40903 } }
I20260619 18:47:30.472263 9379 raft_consensus.cc:374] T 00000000000000000000000000000000 P cd505214823c4932998fdead415c475f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:47:30.472358 9379 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd505214823c4932998fdead415c475f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd505214823c4932998fdead415c475f, State: Initialized, Role: FOLLOWER
I20260619 18:47:30.472890 9379 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd505214823c4932998fdead415c475f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd505214823c4932998fdead415c475f" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 40903 } }
I20260619 18:47:30.473170 9379 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd505214823c4932998fdead415c475f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:47:30.473318 9379 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd505214823c4932998fdead415c475f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:47:30.473500 9379 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd505214823c4932998fdead415c475f [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:47:30.475615 9379 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd505214823c4932998fdead415c475f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd505214823c4932998fdead415c475f" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 40903 } }
I20260619 18:47:30.476034 9379 leader_election.cc:302] T 00000000000000000000000000000000 P cd505214823c4932998fdead415c475f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd505214823c4932998fdead415c475f; no voters:
I20260619 18:47:30.476608 9379 leader_election.cc:288] T 00000000000000000000000000000000 P cd505214823c4932998fdead415c475f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:47:30.476729 9384 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd505214823c4932998fdead415c475f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:47:30.477332 9384 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd505214823c4932998fdead415c475f [term 1 LEADER]: Becoming Leader. State: Replica: cd505214823c4932998fdead415c475f, State: Running, Role: LEADER
I20260619 18:47:30.477802 9384 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd505214823c4932998fdead415c475f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd505214823c4932998fdead415c475f" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 40903 } }
I20260619 18:47:30.478458 9379 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd505214823c4932998fdead415c475f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:47:30.480290 9386 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd505214823c4932998fdead415c475f [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd505214823c4932998fdead415c475f. Latest consensus state: current_term: 1 leader_uuid: "cd505214823c4932998fdead415c475f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd505214823c4932998fdead415c475f" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 40903 } } }
I20260619 18:47:30.480327 9385 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd505214823c4932998fdead415c475f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cd505214823c4932998fdead415c475f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd505214823c4932998fdead415c475f" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 40903 } } }
I20260619 18:47:30.480703 9386 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd505214823c4932998fdead415c475f [sys.catalog]: This master's current role is: LEADER
I20260619 18:47:30.480703 9385 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd505214823c4932998fdead415c475f [sys.catalog]: This master's current role is: LEADER
I20260619 18:47:30.485410 9389 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:47:30.489730 9389 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:47:30.497452 9389 catalog_manager.cc:1347] Generated new cluster ID: 586b044f9c234f13bef3f7d8d6077ddb
I20260619 18:47:30.497552 9389 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:47:30.533239 9389 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:47:30.534655 9389 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:47:30.541699 9389 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cd505214823c4932998fdead415c475f: Generated new TSK 0
I20260619 18:47:30.542445 9389 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 18:47:30.576637 9409 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260619 18:47:30.576740 9409 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 18:47:30.577862 9382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:30.558595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41280 (local address 127.3.243.126:40903)
0619 18:47:30.558778 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:30.558784 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:30.559243 (+ 459us) server_negotiation.cc:407] Connection header received
0619 18:47:30.559433 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:30.559439 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:30.559529 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:30.559680 (+ 151us) server_negotiation.cc:226] Negotiated authn=JWT
0619 18:47:30.560057 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:30.561187 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:30.562451 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:30.563122 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:30.563260 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:30.563670 (+ 410us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 18:47:30.577263 (+ 13593us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36923/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 18:47:30.577580 (+ 317us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41280: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36923/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":55}
W20260619 18:47:30.577965 9408 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:47:30.558599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:40903 (local address 127.0.0.1:41280)
0619 18:47:30.559154 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:30.559170 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:30.559282 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:30.559683 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:30.559687 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:30.559696 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0619 18:47:30.559917 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:30.559924 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:30.561302 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:30.561306 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:30.562326 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:30.562336 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:30.563290 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:30.563294 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:30.563414 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:30.563444 (+ 30us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 18:47:30.577466 (+ 14022us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36923/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 18:47:30.577665 (+ 199us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.243.126:40903: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36923/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":308,"threads_started":1}
W20260619 18:47:30.577980 9382 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41280: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36923/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260619 18:47:30.579258 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 9313
I20260619 18:47:30.580372 9374 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:47:30.673542 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 9313
2026-06-19T18:47:30Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (700 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260619 18:47:30.724961 4045 mini_oidc.cc:152] Starting JWKS server
I20260619 18:47:30.726768 4045 webserver.cc:460] Webserver started at https://127.0.0.1:37807/ using document root <none> and password file <none>
I20260619 18:47:30.726981 4045 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 18:47:30.727563 4045 webserver.cc:460] Webserver started at http://127.0.0.1:42577/ using document root <none> and password file <none>
2026-06-19T18:47:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:47:30Z Disabled control of system clock
I20260619 18:47:30.750982 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781894810715282-4045-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781894810715282-4045-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781894810715282-4045-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.126:41597
--webserver_interface=127.3.243.126
--webserver_port=0
--builtin_ntp_servers=127.3.243.84:39499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.126:41597
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37807/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 18:47:31.050658 9416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:47:31.050948 9416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:47:31.051077 9416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:47:31.058044 9416 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:47:31.058147 9416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:47:31.058208 9416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:47:31.058288 9416 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:47:31.058350 9416 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 18:47:31.058431 9416 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 18:47:31.058485 9416 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37807/jwks.json
I20260619 18:47:31.067255 9416 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:39499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781894810715282-4045-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781894810715282-4045-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.243.126:41597
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.126:41597
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37807/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.243.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781894810715282-4045-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:47:31.068456 9416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:47:31.070134 9416 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:47:31.080819 9421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:31.080865 9422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:31.081264 9424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:47:31.081701 9416 server_base.cc:1034] running on GCE node
I20260619 18:47:31.082360 9416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:47:31.083580 9416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:47:31.084781 9416 hybrid_clock.cc:648] HybridClock initialized: now 1781894851084716 us; error 41 us; skew 500 ppm
I20260619 18:47:31.085088 9416 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 18:47:31.090986 9416 webserver.cc:460] Webserver started at http://127.3.243.126:41197/ using document root <none> and password file <none>
I20260619 18:47:31.091612 9416 fs_manager.cc:359] Metadata directory not provided
I20260619 18:47:31.091709 9416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:47:31.092010 9416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:47:31.094725 9416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781894810715282-4045-0/minicluster-data/master-0/data/instance:
uuid: "6513298ab4fc44f9b10ce595dabe9827"
format_stamp: "Formatted at 2026-06-19 18:47:31 on dist-test-slave-m9wg"
server_key: "b72fc63f3e6b2b30859f1b22ab7bc598"
server_key_iv: "cec44d7a9ea0cb1e1c759833244def54"
server_key_version: "encryptionkey@0"
I20260619 18:47:31.095479 9416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781894810715282-4045-0/minicluster-data/master-0/wal/instance:
uuid: "6513298ab4fc44f9b10ce595dabe9827"
format_stamp: "Formatted at 2026-06-19 18:47:31 on dist-test-slave-m9wg"
server_key: "b72fc63f3e6b2b30859f1b22ab7bc598"
server_key_iv: "cec44d7a9ea0cb1e1c759833244def54"
server_key_version: "encryptionkey@0"
I20260619 18:47:31.100291 9416 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.002s
I20260619 18:47:31.103459 9430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:31.104692 9416 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 18:47:31.104851 9416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781894810715282-4045-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781894810715282-4045-0/minicluster-data/master-0/wal
uuid: "6513298ab4fc44f9b10ce595dabe9827"
format_stamp: "Formatted at 2026-06-19 18:47:31 on dist-test-slave-m9wg"
server_key: "b72fc63f3e6b2b30859f1b22ab7bc598"
server_key_iv: "cec44d7a9ea0cb1e1c759833244def54"
server_key_version: "encryptionkey@0"
I20260619 18:47:31.105031 9416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781894810715282-4045-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781894810715282-4045-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781894810715282-4045-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:47:31.125989 9416 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:47:31.127024 9416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:47:31.127336 9416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:47:31.147831 9416 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.126:41597
I20260619 18:47:31.147848 9481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.126:41597 every 8 connection(s)
I20260619 18:47:31.149437 9416 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
I20260619 18:47:31.153808 9482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:47:31.155089 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 9416
I20260619 18:47:31.155304 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781894810715282-4045-0/minicluster-data/master-0/wal/instance
I20260619 18:47:31.156121 4045 external_mini_cluster.cc:1428] Setting key 9d05ec151441011aafb531088151efb2
I20260619 18:47:31.166476 9482 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6513298ab4fc44f9b10ce595dabe9827: Bootstrap starting.
I20260619 18:47:31.170166 9482 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6513298ab4fc44f9b10ce595dabe9827: Neither blocks nor log segments found. Creating new log.
I20260619 18:47:31.171502 9482 log.cc:824] T 00000000000000000000000000000000 P 6513298ab4fc44f9b10ce595dabe9827: Log is configured to *not* fsync() on all Append() calls
I20260619 18:47:31.174458 9482 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6513298ab4fc44f9b10ce595dabe9827: No bootstrap required, opened a new log
I20260619 18:47:31.174530 9485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:31.161093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47446 (local address 127.3.243.126:41597)
0619 18:47:31.162250 (+ 1157us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:31.162268 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:31.162322 (+ 54us) server_negotiation.cc:407] Connection header received
0619 18:47:31.164815 (+ 2493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:31.164844 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:31.165457 (+ 613us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:31.165950 (+ 493us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:31.166539 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:31.168577 (+ 2038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:31.169886 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:31.170955 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:31.171141 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:31.172212 (+ 1071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:31.172223 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:31.172246 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:47:31.172800 (+ 554us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:31.172887 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:47:31.172907 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:31.173008 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:31.173233 (+ 225us) server_negotiation.cc:299] Negotiation successful
0619 18:47:31.173553 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":344,"threads_started":1}
I20260619 18:47:31.178303 4045 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:47:31.182142 9482 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6513298ab4fc44f9b10ce595dabe9827 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6513298ab4fc44f9b10ce595dabe9827" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 41597 } }
I20260619 18:47:31.182479 9482 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6513298ab4fc44f9b10ce595dabe9827 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:47:31.182572 9482 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6513298ab4fc44f9b10ce595dabe9827 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6513298ab4fc44f9b10ce595dabe9827, State: Initialized, Role: FOLLOWER
I20260619 18:47:31.183089 9482 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6513298ab4fc44f9b10ce595dabe9827 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6513298ab4fc44f9b10ce595dabe9827" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 41597 } }
I20260619 18:47:31.183306 9482 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6513298ab4fc44f9b10ce595dabe9827 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:47:31.183430 9482 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6513298ab4fc44f9b10ce595dabe9827 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:47:31.183580 9482 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6513298ab4fc44f9b10ce595dabe9827 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:47:31.185817 9482 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6513298ab4fc44f9b10ce595dabe9827 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6513298ab4fc44f9b10ce595dabe9827" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 41597 } }
I20260619 18:47:31.186280 9482 leader_election.cc:302] T 00000000000000000000000000000000 P 6513298ab4fc44f9b10ce595dabe9827 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6513298ab4fc44f9b10ce595dabe9827; no voters:
I20260619 18:47:31.186826 9482 leader_election.cc:288] T 00000000000000000000000000000000 P 6513298ab4fc44f9b10ce595dabe9827 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:47:31.186934 9487 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6513298ab4fc44f9b10ce595dabe9827 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:47:31.187927 9487 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6513298ab4fc44f9b10ce595dabe9827 [term 1 LEADER]: Becoming Leader. State: Replica: 6513298ab4fc44f9b10ce595dabe9827, State: Running, Role: LEADER
I20260619 18:47:31.188422 9487 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6513298ab4fc44f9b10ce595dabe9827 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6513298ab4fc44f9b10ce595dabe9827" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 41597 } }
I20260619 18:47:31.188850 9482 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6513298ab4fc44f9b10ce595dabe9827 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:47:31.190922 9488 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6513298ab4fc44f9b10ce595dabe9827 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6513298ab4fc44f9b10ce595dabe9827" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6513298ab4fc44f9b10ce595dabe9827" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 41597 } } }
I20260619 18:47:31.191334 9488 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6513298ab4fc44f9b10ce595dabe9827 [sys.catalog]: This master's current role is: LEADER
I20260619 18:47:31.192131 9496 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:47:31.191955 9489 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6513298ab4fc44f9b10ce595dabe9827 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6513298ab4fc44f9b10ce595dabe9827. Latest consensus state: current_term: 1 leader_uuid: "6513298ab4fc44f9b10ce595dabe9827" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6513298ab4fc44f9b10ce595dabe9827" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 41597 } } }
I20260619 18:47:31.192390 9489 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6513298ab4fc44f9b10ce595dabe9827 [sys.catalog]: This master's current role is: LEADER
I20260619 18:47:31.197584 9496 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:47:31.206660 9496 catalog_manager.cc:1347] Generated new cluster ID: 4907cb3b8e0b4b4daf77a1d9e97fca43
I20260619 18:47:31.206758 9496 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:47:31.219355 9496 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:47:31.220494 9496 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:47:31.227265 9496 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6513298ab4fc44f9b10ce595dabe9827: Generated new TSK 0
I20260619 18:47:31.227953 9496 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:47:31.252338 9485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:31.244752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47454 (local address 127.3.243.126:41597)
0619 18:47:31.244893 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:31.244899 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:31.245355 (+ 456us) server_negotiation.cc:407] Connection header received
0619 18:47:31.245564 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:31.245568 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:31.245644 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:31.245767 (+ 123us) server_negotiation.cc:226] Negotiated authn=JWT
0619 18:47:31.246203 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:31.246941 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:31.249004 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:31.249467 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:31.249559 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:31.250082 (+ 523us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 18:47:31.251722 (+ 1640us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37807/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 18:47:31.252010 (+ 288us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47454: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37807/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":47}
W20260619 18:47:31.252525 9511 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:47:31.244705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:41597 (local address 127.0.0.1:47454)
0619 18:47:31.245261 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:31.245275 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:31.245405 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:31.245797 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:31.245802 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:31.245815 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0619 18:47:31.246065 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:31.246073 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:31.247132 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:31.247138 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:31.248859 (+ 1721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:31.248875 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:31.249639 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:31.249646 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:31.249837 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:31.249881 (+ 44us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 18:47:31.251896 (+ 2015us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37807/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 18:47:31.252116 (+ 220us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.243.126:41597: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37807/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":294,"threads_started":1}
W20260619 18:47:31.252800 9485 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47454: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37807/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
I20260619 18:47:31.253552 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 9416
I20260619 18:47:31.255570 9477 generic_service.cc:196] Checking for leaks (request via RPC)
E20260619 18:47:31.264973 9512 webserver.cc:533] Webserver: sslize failed:
E20260619 18:47:31.265061 9512 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 18:47:31.345573 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 9416
2026-06-19T18:47:31Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (672 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-19T18:47:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:47:31Z Disabled control of system clock
I20260619 18:47:31.419081 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781894810715282-4045-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781894810715282-4045-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781894810715282-4045-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.126:43091
--webserver_interface=127.3.243.126
--webserver_port=0
--builtin_ntp_servers=127.3.243.84:40241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.126:43091
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781894810715282-4045-0/testchainca.cert with env {}
W20260619 18:47:31.718483 9517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:47:31.718745 9517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:47:31.718827 9517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:47:31.725781 9517 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:47:31.725875 9517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:47:31.725911 9517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:47:31.725965 9517 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:47:31.726007 9517 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 18:47:31.726063 9517 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 18:47:31.726102 9517 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260619 18:47:31.735576 9517 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:40241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781894810715282-4045-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781894810715282-4045-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.243.126:43091
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.126:43091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.243.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781894810715282-4045-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781894810715282-4045-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:47:31.736811 9517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:47:31.738018 9517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:47:31.749032 9522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:31.749073 9525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:31.749094 9523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:47:31.749789 9517 server_base.cc:1034] running on GCE node
I20260619 18:47:31.750473 9517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:47:31.751775 9517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:47:31.752970 9517 hybrid_clock.cc:648] HybridClock initialized: now 1781894851752912 us; error 48 us; skew 500 ppm
I20260619 18:47:31.753295 9517 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 18:47:31.755393 9517 webserver.cc:460] Webserver started at http://127.3.243.126:34541/ using document root <none> and password file <none>
I20260619 18:47:31.756055 9517 fs_manager.cc:359] Metadata directory not provided
I20260619 18:47:31.756189 9517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:47:31.756489 9517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:47:31.759179 9517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781894810715282-4045-0/minicluster-data/master-0/data/instance:
uuid: "742ade32f6d54943a405069be4d14511"
format_stamp: "Formatted at 2026-06-19 18:47:31 on dist-test-slave-m9wg"
server_key: "fb2c04a33c1f2b978484a79cc173726a"
server_key_iv: "6e8dfa689adbdc1cd591829c9fc9574e"
server_key_version: "encryptionkey@0"
I20260619 18:47:31.759997 9517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781894810715282-4045-0/minicluster-data/master-0/wal/instance:
uuid: "742ade32f6d54943a405069be4d14511"
format_stamp: "Formatted at 2026-06-19 18:47:31 on dist-test-slave-m9wg"
server_key: "fb2c04a33c1f2b978484a79cc173726a"
server_key_iv: "6e8dfa689adbdc1cd591829c9fc9574e"
server_key_version: "encryptionkey@0"
I20260619 18:47:31.764675 9517 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 18:47:31.767760 9531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:31.769052 9517 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:47:31.769207 9517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781894810715282-4045-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781894810715282-4045-0/minicluster-data/master-0/wal
uuid: "742ade32f6d54943a405069be4d14511"
format_stamp: "Formatted at 2026-06-19 18:47:31 on dist-test-slave-m9wg"
server_key: "fb2c04a33c1f2b978484a79cc173726a"
server_key_iv: "6e8dfa689adbdc1cd591829c9fc9574e"
server_key_version: "encryptionkey@0"
I20260619 18:47:31.769385 9517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781894810715282-4045-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781894810715282-4045-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781894810715282-4045-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:47:31.786087 9517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:47:31.786795 9517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:47:31.787032 9517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:47:31.802901 9517 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.126:43091
I20260619 18:47:31.802906 9582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.126:43091 every 8 connection(s)
I20260619 18:47:31.804100 9517 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
I20260619 18:47:31.807569 9583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:47:31.813526 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 9517
I20260619 18:47:31.813694 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781894810715282-4045-0/minicluster-data/master-0/wal/instance
I20260619 18:47:31.814285 4045 external_mini_cluster.cc:1428] Setting key d1062e89163501bdaeae8db6eb595840
I20260619 18:47:31.817149 9583 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 742ade32f6d54943a405069be4d14511: Bootstrap starting.
I20260619 18:47:31.820230 9583 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 742ade32f6d54943a405069be4d14511: Neither blocks nor log segments found. Creating new log.
I20260619 18:47:31.821183 9583 log.cc:824] T 00000000000000000000000000000000 P 742ade32f6d54943a405069be4d14511: Log is configured to *not* fsync() on all Append() calls
I20260619 18:47:31.823884 9583 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 742ade32f6d54943a405069be4d14511: No bootstrap required, opened a new log
I20260619 18:47:31.826814 9586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:31.815973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43304 (local address 127.3.243.126:43091)
0619 18:47:31.816944 (+ 971us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:31.816954 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:31.816991 (+ 37us) server_negotiation.cc:407] Connection header received
0619 18:47:31.818592 (+ 1601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:31.818617 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:31.819042 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:31.819346 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:31.819893 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:31.821275 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:31.822740 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:31.823366 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:31.823495 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:31.824642 (+ 1147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:31.824650 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:31.824663 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:47:31.825153 (+ 490us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:31.825223 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:47:31.825243 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:31.825321 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:31.825587 (+ 266us) server_negotiation.cc:299] Negotiation successful
0619 18:47:31.825895 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":362,"threads_started":1}
I20260619 18:47:31.830461 4045 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:47:31.832064 9583 raft_consensus.cc:348] T 00000000000000000000000000000000 P 742ade32f6d54943a405069be4d14511 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "742ade32f6d54943a405069be4d14511" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 43091 } }
I20260619 18:47:31.832417 9583 raft_consensus.cc:374] T 00000000000000000000000000000000 P 742ade32f6d54943a405069be4d14511 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:47:31.832516 9583 raft_consensus.cc:729] T 00000000000000000000000000000000 P 742ade32f6d54943a405069be4d14511 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 742ade32f6d54943a405069be4d14511, State: Initialized, Role: FOLLOWER
I20260619 18:47:31.833089 9583 consensus_queue.cc:260] T 00000000000000000000000000000000 P 742ade32f6d54943a405069be4d14511 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "742ade32f6d54943a405069be4d14511" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 43091 } }
I20260619 18:47:31.833297 9583 raft_consensus.cc:388] T 00000000000000000000000000000000 P 742ade32f6d54943a405069be4d14511 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:47:31.833421 9583 raft_consensus.cc:482] T 00000000000000000000000000000000 P 742ade32f6d54943a405069be4d14511 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:47:31.833588 9583 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 742ade32f6d54943a405069be4d14511 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:47:31.835630 9583 raft_consensus.cc:504] T 00000000000000000000000000000000 P 742ade32f6d54943a405069be4d14511 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "742ade32f6d54943a405069be4d14511" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 43091 } }
I20260619 18:47:31.836143 9583 leader_election.cc:302] T 00000000000000000000000000000000 P 742ade32f6d54943a405069be4d14511 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 742ade32f6d54943a405069be4d14511; no voters:
I20260619 18:47:31.836876 9583 leader_election.cc:288] T 00000000000000000000000000000000 P 742ade32f6d54943a405069be4d14511 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:47:31.836961 9588 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 742ade32f6d54943a405069be4d14511 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:47:31.837595 9588 raft_consensus.cc:686] T 00000000000000000000000000000000 P 742ade32f6d54943a405069be4d14511 [term 1 LEADER]: Becoming Leader. State: Replica: 742ade32f6d54943a405069be4d14511, State: Running, Role: LEADER
I20260619 18:47:31.838085 9588 consensus_queue.cc:237] T 00000000000000000000000000000000 P 742ade32f6d54943a405069be4d14511 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "742ade32f6d54943a405069be4d14511" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 43091 } }
I20260619 18:47:31.838784 9583 sys_catalog.cc:564] T 00000000000000000000000000000000 P 742ade32f6d54943a405069be4d14511 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:47:31.840699 9589 sys_catalog.cc:455] T 00000000000000000000000000000000 P 742ade32f6d54943a405069be4d14511 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "742ade32f6d54943a405069be4d14511" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "742ade32f6d54943a405069be4d14511" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 43091 } } }
I20260619 18:47:31.840791 9590 sys_catalog.cc:455] T 00000000000000000000000000000000 P 742ade32f6d54943a405069be4d14511 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 742ade32f6d54943a405069be4d14511. Latest consensus state: current_term: 1 leader_uuid: "742ade32f6d54943a405069be4d14511" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "742ade32f6d54943a405069be4d14511" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 43091 } } }
I20260619 18:47:31.841087 9589 sys_catalog.cc:458] T 00000000000000000000000000000000 P 742ade32f6d54943a405069be4d14511 [sys.catalog]: This master's current role is: LEADER
I20260619 18:47:31.841122 9590 sys_catalog.cc:458] T 00000000000000000000000000000000 P 742ade32f6d54943a405069be4d14511 [sys.catalog]: This master's current role is: LEADER
I20260619 18:47:31.843057 9596 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:47:31.847589 9596 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:47:31.855027 9596 catalog_manager.cc:1347] Generated new cluster ID: eebe36d498f84204a5cd7072c2724224
I20260619 18:47:31.855114 9596 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:47:31.867271 9596 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:47:31.868908 9596 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:47:31.878469 9596 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 742ade32f6d54943a405069be4d14511: Generated new TSK 0
I20260619 18:47:31.879356 9596 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:47:32.329749 9586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:31.891904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43310 (local address 127.3.243.126:43091)
0619 18:47:31.892076 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:31.892083 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:31.892457 (+ 374us) server_negotiation.cc:407] Connection header received
0619 18:47:31.892633 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:31.892638 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:31.892740 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:31.892900 (+ 160us) server_negotiation.cc:226] Negotiated authn=JWT
0619 18:47:31.893310 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:31.894394 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:31.895719 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:31.896419 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:31.896538 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:31.896954 (+ 416us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 18:47:32.329209 (+432255us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 18:47:32.329511 (+ 302us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43310: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":53}
W20260619 18:47:32.329886 9586 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43310: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260619 18:47:32.330037 9612 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:47:31.891817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:43091 (local address 127.0.0.1:43310)
0619 18:47:31.892368 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:31.892383 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:31.892497 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:31.892914 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:31.892917 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:31.892926 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0619 18:47:31.893178 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:31.893186 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:31.894507 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:31.894510 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:31.895583 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:31.895594 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:31.896562 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:31.896566 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:31.896687 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:31.896717 (+ 30us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 18:47:32.329477 (+432760us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 18:47:32.329722 (+ 245us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.243.126:43091: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":279,"threads_started":1}
I20260619 18:47:32.331077 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 9517
I20260619 18:47:32.332113 9578 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:47:32.431401 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 9517
2026-06-19T18:47:32Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (1066 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (391 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (390 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894810715282-4045-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:47:33 dist-test-slave-m9wg krb5kdc[9627](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:47:33 dist-test-slave-m9wg krb5kdc[9627](info): set up 2 sockets
Jun 19 18:47:33 dist-test-slave-m9wg krb5kdc[9627](info): commencing operation
krb5kdc: starting...
W20260619 18:47:34.205375 4045 mini_kdc.cc:121] Time spent starting KDC: real 0.959s user 0.007s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894810715282-4045-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894810715282-4045-0/krb5kdc/test-user.keytab.
Jun 19 18:47:34 dist-test-slave-m9wg krb5kdc[9627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:47:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:47:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.243.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab.
Entry for principal kudu/127.3.243.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab.
I20260619 18:47:34.381510 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894810715282-4045-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894810715282-4045-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894810715282-4045-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.126:44671
--webserver_interface=127.3.243.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab
--principal=kudu/127.3.243.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:32823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.126:44671
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:47:34.684851 9640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:47:34.685173 9640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:47:34.685305 9640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:47:34.692109 9640 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:47:34.692224 9640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:47:34.692310 9640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:47:34.692386 9640 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:47:34.692446 9640 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:47:34.701522 9640 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:32823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894810715282-4045-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894810715282-4045-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.243.126:44671
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab
--principal=kudu/127.3.243.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.126:44671
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.243.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894810715282-4045-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:47:34.702679 9640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:47:34.703886 9640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:47:34.714596 9646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:34.714776 9648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:34.714627 9645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:47:34.715381 9640 server_base.cc:1034] running on GCE node
I20260619 18:47:34.716090 9640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:47:34.717343 9640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:47:34.718539 9640 hybrid_clock.cc:648] HybridClock initialized: now 1781894854718490 us; error 39 us; skew 500 ppm
I20260619 18:47:34.718853 9640 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:47:34 dist-test-slave-m9wg krb5kdc[9627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894854, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:47:34.721719 9640 init.cc:377] Logged in from keytab as kudu/127.3.243.126@KRBTEST.COM (short username kudu)
I20260619 18:47:34.723029 9640 webserver.cc:460] Webserver started at http://127.3.243.126:43803/ using document root <none> and password file <none>
I20260619 18:47:34.723655 9640 fs_manager.cc:359] Metadata directory not provided
I20260619 18:47:34.723767 9640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:47:34.724040 9640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:47:34.726794 9640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894810715282-4045-0/minicluster-data/master-0/data/instance:
uuid: "1207c0e602bd4a40b846f1207f3101ac"
format_stamp: "Formatted at 2026-06-19 18:47:34 on dist-test-slave-m9wg"
server_key: "8fa083c5a447426365fbb69cb9480f75"
server_key_iv: "d66f15acfaf96aa3f24e720b389d3405"
server_key_version: "encryptionkey@0"
I20260619 18:47:34.727515 9640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894810715282-4045-0/minicluster-data/master-0/wal/instance:
uuid: "1207c0e602bd4a40b846f1207f3101ac"
format_stamp: "Formatted at 2026-06-19 18:47:34 on dist-test-slave-m9wg"
server_key: "8fa083c5a447426365fbb69cb9480f75"
server_key_iv: "d66f15acfaf96aa3f24e720b389d3405"
server_key_version: "encryptionkey@0"
I20260619 18:47:34.732170 9640 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260619 18:47:34.735222 9655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:34.736470 9640 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260619 18:47:34.736611 9640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894810715282-4045-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894810715282-4045-0/minicluster-data/master-0/wal
uuid: "1207c0e602bd4a40b846f1207f3101ac"
format_stamp: "Formatted at 2026-06-19 18:47:34 on dist-test-slave-m9wg"
server_key: "8fa083c5a447426365fbb69cb9480f75"
server_key_iv: "d66f15acfaf96aa3f24e720b389d3405"
server_key_version: "encryptionkey@0"
I20260619 18:47:34.736765 9640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894810715282-4045-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894810715282-4045-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894810715282-4045-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:47:34.753077 9640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:47:34.758193 9640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:47:34.758443 9640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:47:34.773485 9640 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.126:44671
I20260619 18:47:34.773504 9706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.126:44671 every 8 connection(s)
I20260619 18:47:34.774706 9640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
I20260619 18:47:34.777174 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 9640
I20260619 18:47:34.777349 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894810715282-4045-0/minicluster-data/master-0/wal/instance
I20260619 18:47:34.777954 4045 external_mini_cluster.cc:1428] Setting key a58aa9ef8e6d68494fd19cb69362255f
I20260619 18:47:34.778307 9707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:47:34.791803 9707 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1207c0e602bd4a40b846f1207f3101ac: Bootstrap starting.
Jun 19 18:47:34 dist-test-slave-m9wg krb5kdc[9627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.126@KRBTEST.COM
I20260619 18:47:34.795054 9707 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1207c0e602bd4a40b846f1207f3101ac: Neither blocks nor log segments found. Creating new log.
I20260619 18:47:34.796100 9707 log.cc:824] T 00000000000000000000000000000000 P 1207c0e602bd4a40b846f1207f3101ac: Log is configured to *not* fsync() on all Append() calls
I20260619 18:47:34.799098 9707 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1207c0e602bd4a40b846f1207f3101ac: No bootstrap required, opened a new log
I20260619 18:47:34.799819 9710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:34.779766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38262 (local address 127.3.243.126:44671)
0619 18:47:34.780548 (+ 782us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:34.780561 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:34.780607 (+ 46us) server_negotiation.cc:407] Connection header received
0619 18:47:34.782811 (+ 2204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:34.782837 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:34.783329 (+ 492us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:34.783768 (+ 439us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:34.784937 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:34.786899 (+ 1962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:34.788067 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:34.789143 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:34.789346 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:34.793098 (+ 3752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:34.793106 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:34.793120 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:34.793157 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:34.795935 (+ 2778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:34.796769 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:34.796781 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:34.796789 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:34.796985 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:34.797344 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:34.797349 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:34.797351 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:34.797829 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:34.798031 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:34.798484 (+ 453us) server_negotiation.cc:299] Negotiation successful
0619 18:47:34.798803 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":332,"threads_started":1}
I20260619 18:47:34.807371 9707 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1207c0e602bd4a40b846f1207f3101ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1207c0e602bd4a40b846f1207f3101ac" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 44671 } }
I20260619 18:47:34.807718 9707 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1207c0e602bd4a40b846f1207f3101ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:47:34.807808 9707 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1207c0e602bd4a40b846f1207f3101ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1207c0e602bd4a40b846f1207f3101ac, State: Initialized, Role: FOLLOWER
I20260619 18:47:34.808359 9707 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1207c0e602bd4a40b846f1207f3101ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1207c0e602bd4a40b846f1207f3101ac" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 44671 } }
I20260619 18:47:34.808636 9707 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1207c0e602bd4a40b846f1207f3101ac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:47:34.808782 9707 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1207c0e602bd4a40b846f1207f3101ac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:47:34.808961 9707 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1207c0e602bd4a40b846f1207f3101ac [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:47:34.811060 9707 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1207c0e602bd4a40b846f1207f3101ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1207c0e602bd4a40b846f1207f3101ac" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 44671 } }
I20260619 18:47:34.811486 9707 leader_election.cc:302] T 00000000000000000000000000000000 P 1207c0e602bd4a40b846f1207f3101ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1207c0e602bd4a40b846f1207f3101ac; no voters:
I20260619 18:47:34.811964 9707 leader_election.cc:288] T 00000000000000000000000000000000 P 1207c0e602bd4a40b846f1207f3101ac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:47:34.812075 9712 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1207c0e602bd4a40b846f1207f3101ac [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:47:34.812685 9712 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1207c0e602bd4a40b846f1207f3101ac [term 1 LEADER]: Becoming Leader. State: Replica: 1207c0e602bd4a40b846f1207f3101ac, State: Running, Role: LEADER
I20260619 18:47:34.813187 9712 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1207c0e602bd4a40b846f1207f3101ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1207c0e602bd4a40b846f1207f3101ac" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 44671 } }
I20260619 18:47:34.813995 9707 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1207c0e602bd4a40b846f1207f3101ac [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:47:34.815977 9713 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1207c0e602bd4a40b846f1207f3101ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1207c0e602bd4a40b846f1207f3101ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1207c0e602bd4a40b846f1207f3101ac" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 44671 } } }
I20260619 18:47:34.816383 9713 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1207c0e602bd4a40b846f1207f3101ac [sys.catalog]: This master's current role is: LEADER
I20260619 18:47:34.816814 9714 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1207c0e602bd4a40b846f1207f3101ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1207c0e602bd4a40b846f1207f3101ac. Latest consensus state: current_term: 1 leader_uuid: "1207c0e602bd4a40b846f1207f3101ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1207c0e602bd4a40b846f1207f3101ac" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 44671 } } }
I20260619 18:47:34.817094 9714 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1207c0e602bd4a40b846f1207f3101ac [sys.catalog]: This master's current role is: LEADER
I20260619 18:47:34.820863 9720 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:47:34.825683 9720 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:47:34.834316 9720 catalog_manager.cc:1347] Generated new cluster ID: 361e9394bb584cfaa7aadf40a0718a0c
I20260619 18:47:34.834432 9720 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:47:34.847746 9720 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:47:34.849064 9720 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:47:34.855432 9720 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1207c0e602bd4a40b846f1207f3101ac: Generated new TSK 0
I20260619 18:47:34.856243 9720 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.243.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.65.keytab.
Entry for principal kudu/127.3.243.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.65.keytab.
I20260619 18:47:34.907173 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894810715282-4045-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894810715282-4045-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894810715282-4045-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894810715282-4045-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.65:0
--local_ip_for_outbound_sockets=127.3.243.65
--webserver_interface=127.3.243.65
--webserver_port=0
--tserver_master_addrs=127.3.243.126:44671
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.65.keytab
--principal=kudu/127.3.243.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:32823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:47:35.205994 9733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:47:35.206244 9733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:47:35.206358 9733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:47:35.213392 9733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:47:35.213497 9733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:47:35.213704 9733 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.65
I20260619 18:47:35.223682 9733 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:32823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894810715282-4045-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894810715282-4045-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.65.keytab
--principal=kudu/127.3.243.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894810715282-4045-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.243.65
--webserver_port=0
--tserver_master_addrs=127.3.243.126:44671
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.65
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894810715282-4045-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:47:35.224815 9733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:47:35.225997 9733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:47:35.237659 9738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:35.237732 9739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:35.237699 9741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:47:35.238679 9733 server_base.cc:1034] running on GCE node
I20260619 18:47:35.239197 9733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:47:35.240029 9733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:47:35.241209 9733 hybrid_clock.cc:648] HybridClock initialized: now 1781894855241155 us; error 67 us; skew 500 ppm
I20260619 18:47:35.241479 9733 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:47:35 dist-test-slave-m9wg krb5kdc[9627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894855, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:47:35.244292 9733 init.cc:377] Logged in from keytab as kudu/127.3.243.65@KRBTEST.COM (short username kudu)
I20260619 18:47:35.245513 9733 webserver.cc:460] Webserver started at http://127.3.243.65:36877/ using document root <none> and password file <none>
I20260619 18:47:35.246114 9733 fs_manager.cc:359] Metadata directory not provided
I20260619 18:47:35.246222 9733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:47:35.246488 9733 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:47:35.249326 9733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894810715282-4045-0/minicluster-data/ts-0/data/instance:
uuid: "2088dd64719643198428446d79d8bc79"
format_stamp: "Formatted at 2026-06-19 18:47:35 on dist-test-slave-m9wg"
server_key: "34d0b5bd21438e4503aaa62d39723cd2"
server_key_iv: "2a4cbc1a21a0f5e025ece1424ca9f7f3"
server_key_version: "encryptionkey@0"
I20260619 18:47:35.250025 9733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894810715282-4045-0/minicluster-data/ts-0/wal/instance:
uuid: "2088dd64719643198428446d79d8bc79"
format_stamp: "Formatted at 2026-06-19 18:47:35 on dist-test-slave-m9wg"
server_key: "34d0b5bd21438e4503aaa62d39723cd2"
server_key_iv: "2a4cbc1a21a0f5e025ece1424ca9f7f3"
server_key_version: "encryptionkey@0"
I20260619 18:47:35.254549 9733 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260619 18:47:35.257755 9748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:35.258894 9733 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 18:47:35.259025 9733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894810715282-4045-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894810715282-4045-0/minicluster-data/ts-0/wal
uuid: "2088dd64719643198428446d79d8bc79"
format_stamp: "Formatted at 2026-06-19 18:47:35 on dist-test-slave-m9wg"
server_key: "34d0b5bd21438e4503aaa62d39723cd2"
server_key_iv: "2a4cbc1a21a0f5e025ece1424ca9f7f3"
server_key_version: "encryptionkey@0"
I20260619 18:47:35.259187 9733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894810715282-4045-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894810715282-4045-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894810715282-4045-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:47:35.276352 9733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:47:35.279750 9733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:47:35.279979 9733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:47:35.280859 9733 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:47:35.282090 9733 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:47:35.282162 9733 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:35.282275 9733 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:47:35.282326 9733 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:35.310556 9733 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.65:36177
I20260619 18:47:35.310595 9860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.65:36177 every 8 connection(s)
I20260619 18:47:35.311729 9733 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894810715282-4045-0/minicluster-data/ts-0/data/info.pb
I20260619 18:47:35.313158 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 9733
I20260619 18:47:35.313345 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894810715282-4045-0/minicluster-data/ts-0/wal/instance
I20260619 18:47:35.314154 4045 external_mini_cluster.cc:1428] Setting key 1efa9f970b69a46f29808c07135816f8
Jun 19 18:47:35 dist-test-slave-m9wg krb5kdc[9627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894855, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.65@KRBTEST.COM for kudu/127.3.243.126@KRBTEST.COM
I20260619 18:47:35.332015 9863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:35.314080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.65:32807 (local address 127.3.243.126:44671)
0619 18:47:35.314466 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:35.314470 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:35.315653 (+ 1183us) server_negotiation.cc:407] Connection header received
0619 18:47:35.317856 (+ 2203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:35.317861 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:35.317963 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:35.318091 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:35.319578 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:35.320570 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:35.322582 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:35.323181 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:35.323279 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:35.326725 (+ 3446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:35.326734 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:35.326737 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:35.326774 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:35.329234 (+ 2460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:35.329892 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:35.329898 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:35.329901 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:35.330014 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:35.330421 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:35.330426 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:35.330428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:35.330685 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:35.330837 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:35.331445 (+ 608us) server_negotiation.cc:299] Negotiation successful
0619 18:47:35.331657 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":194,"threads_started":1}
WARNING: no policy specified for kudu/127.3.243.66@KRBTEST.COM; defaulting to no policy
I20260619 18:47:35.332924 9864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:35.314744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:44671 (local address 127.3.243.65:32807)
0619 18:47:35.315505 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:35.315551 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:35.317220 (+ 1669us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:35.318351 (+ 1131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:35.318360 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:35.318864 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:47:35.319412 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:35.319429 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:35.320747 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:35.320753 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:35.322436 (+ 1683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:35.322448 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:35.323392 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:35.323399 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:35.323657 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:35.324315 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:47:35.324343 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:47:35.326490 (+ 2147us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:47:35.329362 (+ 2872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:35.329370 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:35.329386 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:35.329777 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:35.330115 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:35.330120 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:35.330124 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:35.330315 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:35.330807 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:47:35.330814 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:47:35.331116 (+ 302us) client_negotiation.cc:769] Sending connection context
0619 18:47:35.331351 (+ 235us) client_negotiation.cc:240] Negotiation successful
0619 18:47:35.331830 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":302,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.66@KRBTEST.COM" created.
I20260619 18:47:35.334342 9861 heartbeater.cc:347] Connected to a master server at 127.3.243.126:44671
I20260619 18:47:35.334622 9861 heartbeater.cc:464] Registering TS with master...
I20260619 18:47:35.335486 9861 heartbeater.cc:511] Master 127.3.243.126:44671 requested a full tablet report, sending...
I20260619 18:47:35.337579 9672 ts_manager.cc:194] Registered new tserver with Master: 2088dd64719643198428446d79d8bc79 (127.3.243.65:36177)
I20260619 18:47:35.338958 9672 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.65@KRBTEST.COM'} at 127.3.243.65:32807
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.66.keytab.
Entry for principal kudu/127.3.243.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.66.keytab.
I20260619 18:47:35.355135 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894810715282-4045-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894810715282-4045-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894810715282-4045-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894810715282-4045-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.66:0
--local_ip_for_outbound_sockets=127.3.243.66
--webserver_interface=127.3.243.66
--webserver_port=0
--tserver_master_addrs=127.3.243.126:44671
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.66.keytab
--principal=kudu/127.3.243.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:32823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:47:35.653674 9867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:47:35.653959 9867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:47:35.654086 9867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:47:35.661167 9867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:47:35.661294 9867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:47:35.661443 9867 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.66
I20260619 18:47:35.670879 9867 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:32823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894810715282-4045-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894810715282-4045-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.66.keytab
--principal=kudu/127.3.243.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894810715282-4045-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.243.66
--webserver_port=0
--tserver_master_addrs=127.3.243.126:44671
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.66
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894810715282-4045-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:47:35.672117 9867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:47:35.673404 9867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:47:35.685564 9872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:35.685674 9875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:35.685729 9873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:47:35.686555 9867 server_base.cc:1034] running on GCE node
I20260619 18:47:35.687067 9867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:47:35.687911 9867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:47:35.689134 9867 hybrid_clock.cc:648] HybridClock initialized: now 1781894855689066 us; error 54 us; skew 500 ppm
I20260619 18:47:35.689440 9867 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:47:35 dist-test-slave-m9wg krb5kdc[9627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894855, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:47:35.692358 9867 init.cc:377] Logged in from keytab as kudu/127.3.243.66@KRBTEST.COM (short username kudu)
I20260619 18:47:35.693670 9867 webserver.cc:460] Webserver started at http://127.3.243.66:41973/ using document root <none> and password file <none>
I20260619 18:47:35.694299 9867 fs_manager.cc:359] Metadata directory not provided
I20260619 18:47:35.694414 9867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:47:35.694705 9867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:47:35.697589 9867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894810715282-4045-0/minicluster-data/ts-1/data/instance:
uuid: "6a78a049f9874fd28650117004d3764f"
format_stamp: "Formatted at 2026-06-19 18:47:35 on dist-test-slave-m9wg"
server_key: "bb5d9cc1bb30dfff54cdb1ce35d1d794"
server_key_iv: "4b19d9621e044c031638e468e723e323"
server_key_version: "encryptionkey@0"
I20260619 18:47:35.698340 9867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894810715282-4045-0/minicluster-data/ts-1/wal/instance:
uuid: "6a78a049f9874fd28650117004d3764f"
format_stamp: "Formatted at 2026-06-19 18:47:35 on dist-test-slave-m9wg"
server_key: "bb5d9cc1bb30dfff54cdb1ce35d1d794"
server_key_iv: "4b19d9621e044c031638e468e723e323"
server_key_version: "encryptionkey@0"
I20260619 18:47:35.703678 9867 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 18:47:35.707190 9882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:35.708540 9867 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260619 18:47:35.708689 9867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894810715282-4045-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894810715282-4045-0/minicluster-data/ts-1/wal
uuid: "6a78a049f9874fd28650117004d3764f"
format_stamp: "Formatted at 2026-06-19 18:47:35 on dist-test-slave-m9wg"
server_key: "bb5d9cc1bb30dfff54cdb1ce35d1d794"
server_key_iv: "4b19d9621e044c031638e468e723e323"
server_key_version: "encryptionkey@0"
I20260619 18:47:35.708877 9867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894810715282-4045-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894810715282-4045-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894810715282-4045-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:47:35.722960 9867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:47:35.726541 9867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:47:35.726791 9867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:47:35.727664 9867 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:47:35.728897 9867 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:47:35.728979 9867 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:35.729099 9867 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:47:35.729151 9867 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:35.758285 9867 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.66:35203
I20260619 18:47:35.758320 9994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.66:35203 every 8 connection(s)
I20260619 18:47:35.759549 9867 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894810715282-4045-0/minicluster-data/ts-1/data/info.pb
I20260619 18:47:35.760888 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 9867
I20260619 18:47:35.761040 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894810715282-4045-0/minicluster-data/ts-1/wal/instance
I20260619 18:47:35.761469 4045 external_mini_cluster.cc:1428] Setting key 9177b6eb911af5d57ee79be41ffbfdbe
Jun 19 18:47:35 dist-test-slave-m9wg krb5kdc[9627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894855, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.66@KRBTEST.COM for kudu/127.3.243.126@KRBTEST.COM
I20260619 18:47:35.780685 9863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:35.762299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.66:34659 (local address 127.3.243.126:44671)
0619 18:47:35.762569 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:35.762575 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:35.763908 (+ 1333us) server_negotiation.cc:407] Connection header received
0619 18:47:35.765700 (+ 1792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:35.765706 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:35.765800 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:35.765885 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:35.767388 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:35.768080 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:35.770080 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:35.770662 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:35.770783 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:35.774114 (+ 3331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:35.774118 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:35.774122 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:35.774155 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:35.777739 (+ 3584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:35.778429 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:35.778434 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:35.778438 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:35.778588 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:35.779002 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:35.779006 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:35.779008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:35.779287 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:35.779435 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:35.780084 (+ 649us) server_negotiation.cc:299] Negotiation successful
0619 18:47:35.780355 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125}
I20260619 18:47:35.781715 9997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:35.762927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:44671 (local address 127.3.243.66:34659)
0619 18:47:35.763757 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:35.763796 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:35.765426 (+ 1630us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:35.766138 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:35.766147 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:35.766661 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:47:35.767220 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:35.767238 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:35.768260 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:35.768263 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:35.769943 (+ 1680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:35.769956 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:35.770894 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:35.770899 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:35.771105 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:35.771729 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:47:35.771749 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:47:35.773910 (+ 2161us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:47:35.777878 (+ 3968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:35.777887 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:35.777904 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:35.778313 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:35.778692 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:35.778697 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:35.778701 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:35.778895 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:35.779410 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:47:35.779417 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:47:35.779752 (+ 335us) client_negotiation.cc:769] Sending connection context
0619 18:47:35.779994 (+ 242us) client_negotiation.cc:240] Negotiation successful
0619 18:47:35.780553 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":324,"threads_started":1}
WARNING: no policy specified for kudu/127.3.243.67@KRBTEST.COM; defaulting to no policy
I20260619 18:47:35.783139 9995 heartbeater.cc:347] Connected to a master server at 127.3.243.126:44671
I20260619 18:47:35.783448 9995 heartbeater.cc:464] Registering TS with master...
I20260619 18:47:35.784392 9995 heartbeater.cc:511] Master 127.3.243.126:44671 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.67@KRBTEST.COM" created.
I20260619 18:47:35.785996 9672 ts_manager.cc:194] Registered new tserver with Master: 6a78a049f9874fd28650117004d3764f (127.3.243.66:35203)
I20260619 18:47:35.787037 9672 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.66@KRBTEST.COM'} at 127.3.243.66:34659
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.67.keytab.
Entry for principal kudu/127.3.243.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.67.keytab.
I20260619 18:47:35.806998 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894810715282-4045-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894810715282-4045-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894810715282-4045-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894810715282-4045-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.67:0
--local_ip_for_outbound_sockets=127.3.243.67
--webserver_interface=127.3.243.67
--webserver_port=0
--tserver_master_addrs=127.3.243.126:44671
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.67.keytab
--principal=kudu/127.3.243.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:32823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:47:36.106439 10000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:47:36.106707 10000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:47:36.106804 10000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:47:36.113924 10000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:47:36.114027 10000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:47:36.114144 10000 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.67
I20260619 18:47:36.123601 10000 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:32823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894810715282-4045-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894810715282-4045-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.67.keytab
--principal=kudu/127.3.243.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894810715282-4045-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.243.67
--webserver_port=0
--tserver_master_addrs=127.3.243.126:44671
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.67
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894810715282-4045-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:47:36.124867 10000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:47:36.126120 10000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:47:36.138356 10005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:36.138389 10006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:36.138428 10008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:47:36.139055 10000 server_base.cc:1034] running on GCE node
I20260619 18:47:36.139644 10000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:47:36.140729 10000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:47:36.141947 10000 hybrid_clock.cc:648] HybridClock initialized: now 1781894856141898 us; error 47 us; skew 500 ppm
I20260619 18:47:36.142258 10000 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:47:36 dist-test-slave-m9wg krb5kdc[9627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894856, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:47:36.145152 10000 init.cc:377] Logged in from keytab as kudu/127.3.243.67@KRBTEST.COM (short username kudu)
I20260619 18:47:36.146488 10000 webserver.cc:460] Webserver started at http://127.3.243.67:36939/ using document root <none> and password file <none>
I20260619 18:47:36.147148 10000 fs_manager.cc:359] Metadata directory not provided
I20260619 18:47:36.147264 10000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:47:36.147540 10000 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:47:36.150439 10000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894810715282-4045-0/minicluster-data/ts-2/data/instance:
uuid: "6a4bb40192a7497f84fa7be2b5e61daf"
format_stamp: "Formatted at 2026-06-19 18:47:36 on dist-test-slave-m9wg"
server_key: "86387699c37526f9165d8a19ee555394"
server_key_iv: "693940bc7a8bc28c8564c4ea37bdefae"
server_key_version: "encryptionkey@0"
I20260619 18:47:36.151183 10000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894810715282-4045-0/minicluster-data/ts-2/wal/instance:
uuid: "6a4bb40192a7497f84fa7be2b5e61daf"
format_stamp: "Formatted at 2026-06-19 18:47:36 on dist-test-slave-m9wg"
server_key: "86387699c37526f9165d8a19ee555394"
server_key_iv: "693940bc7a8bc28c8564c4ea37bdefae"
server_key_version: "encryptionkey@0"
I20260619 18:47:36.155951 10000 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.007s
I20260619 18:47:36.159375 10015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:36.160693 10000 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:47:36.160835 10000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894810715282-4045-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894810715282-4045-0/minicluster-data/ts-2/wal
uuid: "6a4bb40192a7497f84fa7be2b5e61daf"
format_stamp: "Formatted at 2026-06-19 18:47:36 on dist-test-slave-m9wg"
server_key: "86387699c37526f9165d8a19ee555394"
server_key_iv: "693940bc7a8bc28c8564c4ea37bdefae"
server_key_version: "encryptionkey@0"
I20260619 18:47:36.161011 10000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894810715282-4045-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894810715282-4045-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894810715282-4045-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:47:36.190662 10000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:47:36.194563 10000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:47:36.194832 10000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:47:36.195744 10000 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:47:36.197044 10000 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:47:36.197127 10000 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:36.197244 10000 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:47:36.197302 10000 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:36.227771 10000 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.67:42177
I20260619 18:47:36.227811 10127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.67:42177 every 8 connection(s)
I20260619 18:47:36.229049 10000 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894810715282-4045-0/minicluster-data/ts-2/data/info.pb
I20260619 18:47:36.233630 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 10000
I20260619 18:47:36.233780 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894810715282-4045-0/minicluster-data/ts-2/wal/instance
I20260619 18:47:36.234295 4045 external_mini_cluster.cc:1428] Setting key ac125cb3e95f0cd33c77a033c47f79be
Jun 19 18:47:36 dist-test-slave-m9wg krb5kdc[9627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894856, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.67@KRBTEST.COM for kudu/127.3.243.126@KRBTEST.COM
I20260619 18:47:36.247581 9863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:36.231456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.67:36401 (local address 127.3.243.126:44671)
0619 18:47:36.231644 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:36.231649 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:36.232803 (+ 1154us) server_negotiation.cc:407] Connection header received
0619 18:47:36.234601 (+ 1798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:36.234608 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:36.234697 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:36.234813 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:36.236621 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:36.237522 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:36.238954 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:36.239332 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:36.239416 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:36.242543 (+ 3127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:36.242548 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:36.242551 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:36.242583 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:36.245033 (+ 2450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:36.245673 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:36.245677 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:36.245679 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:36.245807 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:36.246165 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:36.246169 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:36.246170 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:36.246426 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:36.246529 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:36.247119 (+ 590us) server_negotiation.cc:299] Negotiation successful
0619 18:47:36.247296 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 18:47:36.248648 10130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:36.231949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:44671 (local address 127.3.243.67:36401)
0619 18:47:36.232653 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:36.232690 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:36.234334 (+ 1644us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:36.235038 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:36.235049 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:36.235713 (+ 664us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:47:36.236463 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:36.236488 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:36.237665 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:36.237669 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:36.238847 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:36.238858 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:36.239487 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:36.239492 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:36.239712 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:36.240300 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:47:36.240320 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:47:36.242355 (+ 2035us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:47:36.245152 (+ 2797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:36.245158 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:36.245171 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:36.245579 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:36.245883 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:36.245886 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:36.245889 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:36.246057 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:36.246518 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:47:36.246523 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:47:36.246814 (+ 291us) client_negotiation.cc:769] Sending connection context
0619 18:47:36.247024 (+ 210us) client_negotiation.cc:240] Negotiation successful
0619 18:47:36.247414 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":300,"threads_started":1}
I20260619 18:47:36.249924 10128 heartbeater.cc:347] Connected to a master server at 127.3.243.126:44671
I20260619 18:47:36.250200 10128 heartbeater.cc:464] Registering TS with master...
I20260619 18:47:36.250905 10128 heartbeater.cc:511] Master 127.3.243.126:44671 requested a full tablet report, sending...
I20260619 18:47:36.252205 9672 ts_manager.cc:194] Registered new tserver with Master: 6a4bb40192a7497f84fa7be2b5e61daf (127.3.243.67:42177)
I20260619 18:47:36.252979 9672 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.67@KRBTEST.COM'} at 127.3.243.67:36401
I20260619 18:47:36.261224 4045 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894810715282-4045-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894810715282-4045-0/krb5kdc/test-admin.keytab.
Jun 19 18:47:36 dist-test-slave-m9wg krb5kdc[9627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:47:36.285559 4045 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260619 18:47:36.290019 10135 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:47:36.287909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37306 (local address 127.3.243.65:36177)
0619 18:47:36.288536 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:36.288543 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:36.288570 (+ 27us) server_negotiation.cc:407] Connection header received
0619 18:47:36.288637 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:36.288643 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:36.288861 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:36.288994 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:36.289710 (+ 716us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37306: BlockingRecv error: recv got EOF from 127.0.0.1:37306 (error 108)
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":373,"threads_started":1}
W20260619 18:47:36.290270 4045 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894810715282-4045-0/krb5kdc/krb5cc' of type FILE
Jun 19 18:47:36 dist-test-slave-m9wg krb5kdc[9627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:47:36.291627 4045 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 18:47:36 dist-test-slave-m9wg krb5kdc[9627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:47:36.293012 4045 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 18:47:36 dist-test-slave-m9wg krb5kdc[9627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.65@KRBTEST.COM
I20260619 18:47:36.307222 10135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:36.294445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37312 (local address 127.3.243.65:36177)
0619 18:47:36.294579 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:36.294584 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:36.294925 (+ 341us) server_negotiation.cc:407] Connection header received
0619 18:47:36.295048 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:36.295051 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:36.295108 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:36.295183 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:36.296074 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:36.297107 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:36.298205 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:36.298740 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:36.298832 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:36.302148 (+ 3316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:36.302160 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:36.302173 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:36.302216 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:36.304746 (+ 2530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:36.305350 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:36.305357 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:36.305360 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:36.305479 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:36.305827 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:36.305831 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:36.305833 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:36.306169 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:36.306451 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:36.306727 (+ 276us) server_negotiation.cc:299] Negotiation successful
0619 18:47:36.306970 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260619 18:47:36.312616 10135 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:47:36.311085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37322 (local address 127.3.243.65:36177)
0619 18:47:36.311259 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:36.311265 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:36.311376 (+ 111us) server_negotiation.cc:407] Connection header received
0619 18:47:36.311489 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:36.311494 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:36.311572 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:36.311666 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:36.312415 (+ 749us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37322: BlockingRecv error: recv got EOF from 127.0.0.1:37322 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260619 18:47:36.312875 4045 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894810715282-4045-0/krb5kdc/krb5cc' of type FILE
Jun 19 18:47:36 dist-test-slave-m9wg krb5kdc[9627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:47:36.314374 4045 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 18:47:36 dist-test-slave-m9wg krb5kdc[9627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:47:36.315625 4045 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 18:47:36 dist-test-slave-m9wg krb5kdc[9627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.65@KRBTEST.COM
I20260619 18:47:36.328264 10135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:36.316858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37326 (local address 127.3.243.65:36177)
0619 18:47:36.317005 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:36.317010 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:36.317313 (+ 303us) server_negotiation.cc:407] Connection header received
0619 18:47:36.317420 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:36.317424 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:36.317493 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:36.317582 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:36.318442 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:36.319316 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:36.320480 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:36.320934 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:36.321025 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:36.324129 (+ 3104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:36.324136 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:36.324141 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:36.324195 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:36.326392 (+ 2197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:36.326901 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:36.326905 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:36.326907 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:36.327011 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:36.327267 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:36.327270 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:36.327272 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:36.327496 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:36.327665 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:36.327861 (+ 196us) server_negotiation.cc:299] Negotiation successful
0619 18:47:36.328012 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260619 18:47:36.332114 10135 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:47:36.330552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37332 (local address 127.3.243.65:36177)
0619 18:47:36.330722 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:36.330728 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:36.331004 (+ 276us) server_negotiation.cc:407] Connection header received
0619 18:47:36.331135 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:36.331140 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:36.331214 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:36.331287 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:36.331918 (+ 631us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37332: BlockingRecv error: recv got EOF from 127.0.0.1:37332 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260619 18:47:36.332285 4045 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894810715282-4045-0/krb5kdc/krb5cc' of type FILE
Jun 19 18:47:36 dist-test-slave-m9wg krb5kdc[9627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:47:36.333706 4045 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 18:47:36 dist-test-slave-m9wg krb5kdc[9627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:47:36.334899 4045 init.cc:295] Successfully reacquired a new kerberos TGT
I20260619 18:47:36.341818 9861 heartbeater.cc:503] Master 127.3.243.126:44671 was elected leader, sending a full tablet report...
Jun 19 18:47:36 dist-test-slave-m9wg krb5kdc[9627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.65@KRBTEST.COM
I20260619 18:47:36.347519 10135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:36.335973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37340 (local address 127.3.243.65:36177)
0619 18:47:36.336167 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:36.336173 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:36.336430 (+ 257us) server_negotiation.cc:407] Connection header received
0619 18:47:36.336553 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:36.336558 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:36.336624 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:36.336695 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:36.337575 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:36.338256 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:36.339406 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:36.339794 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:36.339894 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:36.343165 (+ 3271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:36.343171 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:36.343175 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:36.343209 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:36.345462 (+ 2253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:36.345984 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:36.345987 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:36.345989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:36.346127 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:36.346407 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:36.346410 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:36.346412 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:36.346663 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:36.346870 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:36.347123 (+ 253us) server_negotiation.cc:299] Negotiation successful
0619 18:47:36.347278 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 18:47:36.348862 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 9733
I20260619 18:47:36.359004 10135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:36.349495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37356 (local address 127.3.243.65:36177)
0619 18:47:36.349656 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:36.349662 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:36.349912 (+ 250us) server_negotiation.cc:407] Connection header received
0619 18:47:36.350009 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:36.350012 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:36.350062 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:36.350140 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:36.350869 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:36.351511 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:36.352701 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:36.353062 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:36.353162 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:36.354849 (+ 1687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:36.354852 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:36.354855 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:36.354876 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:36.356994 (+ 2118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:36.357538 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:36.357542 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:36.357544 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:36.357664 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:36.357934 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:36.357937 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:36.357939 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:36.358168 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:36.358366 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:36.358555 (+ 189us) server_negotiation.cc:299] Negotiation successful
0619 18:47:36.358695 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 18:47:36.359483 9856 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:47:36.439486 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 9733
I20260619 18:47:36.452428 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 9867
Jun 19 18:47:36 dist-test-slave-m9wg krb5kdc[9627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.66@KRBTEST.COM
I20260619 18:47:36.466413 10148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:36.453289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59974 (local address 127.3.243.66:35203)
0619 18:47:36.453797 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:36.453804 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:36.453829 (+ 25us) server_negotiation.cc:407] Connection header received
0619 18:47:36.453887 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:36.453893 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:36.454093 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:36.454231 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:36.455092 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:36.456099 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:36.457289 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:36.457808 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:36.457901 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:36.461413 (+ 3512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:36.461424 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:36.461437 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:36.461467 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:36.463853 (+ 2386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:36.464498 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:36.464506 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:36.464508 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:36.464622 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:36.464952 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:36.464956 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:36.464960 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:36.465282 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:36.465584 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:36.465874 (+ 290us) server_negotiation.cc:299] Negotiation successful
0619 18:47:36.466111 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":254,"threads_started":1}
I20260619 18:47:36.467660 9990 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:47:36.547531 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 9867
I20260619 18:47:36.559888 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 10000
Jun 19 18:47:36 dist-test-slave-m9wg krb5kdc[9627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.67@KRBTEST.COM
I20260619 18:47:36.574056 10150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:36.560774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58186 (local address 127.3.243.67:42177)
0619 18:47:36.561381 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:36.561388 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:36.561417 (+ 29us) server_negotiation.cc:407] Connection header received
0619 18:47:36.561477 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:36.561483 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:36.561702 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:36.561827 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:36.562718 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:36.563767 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:36.565049 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:36.565654 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:36.565789 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:36.569492 (+ 3703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:36.569503 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:36.569517 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:36.569547 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:36.571887 (+ 2340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:36.572489 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:36.572495 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:36.572498 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:36.572608 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:36.572907 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:36.572910 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:36.572912 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:36.573219 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:36.573436 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:36.573608 (+ 172us) server_negotiation.cc:299] Negotiation successful
0619 18:47:36.573787 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":309,"threads_started":1}
I20260619 18:47:36.575343 10123 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:47:36.657258 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 10000
I20260619 18:47:36.669337 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 9640
I20260619 18:47:36.670445 9702 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:47:36.770835 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 9640
2026-06-19T18:47:36Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3560 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:47:36 dist-test-slave-m9wg krb5kdc[10155](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:47:36 dist-test-slave-m9wg krb5kdc[10155](info): set up 2 sockets
Jun 19 18:47:36 dist-test-slave-m9wg krb5kdc[10155](info): commencing operation
krb5kdc: starting...
W20260619 18:47:37.768523 4045 mini_kdc.cc:121] Time spent starting KDC: real 0.962s user 0.001s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/krb5kdc/test-user.keytab.
Jun 19 18:47:37 dist-test-slave-m9wg krb5kdc[10155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894857, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:47:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:47:37Z Disabled control of system clock
WARNING: no policy specified for oryx/127.3.243.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.243.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.243.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/krb5kdc/oryx_127.3.243.126.keytab.
Entry for principal oryx/127.3.243.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/krb5kdc/oryx_127.3.243.126.keytab.
I20260619 18:47:37.953208 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.126:41477
--webserver_interface=127.3.243.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/krb5kdc/oryx_127.3.243.126.keytab
--principal=oryx/127.3.243.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:37789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.126:41477
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:47:38.256698 10168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:47:38.256950 10168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:47:38.257071 10168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:47:38.263962 10168 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:47:38.264050 10168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:47:38.264120 10168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:47:38.264211 10168 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:47:38.264266 10168 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:47:38.273384 10168 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:37789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.243.126:41477
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/krb5kdc/oryx_127.3.243.126.keytab
--principal=oryx/127.3.243.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.126:41477
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.243.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:47:38.274559 10168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:47:38.275767 10168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:47:38.286378 10173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:38.286439 10176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:38.286434 10174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:47:38.287460 10168 server_base.cc:1034] running on GCE node
I20260619 18:47:38.288115 10168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:47:38.289458 10168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:47:38.290632 10168 hybrid_clock.cc:648] HybridClock initialized: now 1781894858290575 us; error 49 us; skew 500 ppm
I20260619 18:47:38.290931 10168 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:47:38 dist-test-slave-m9wg krb5kdc[10155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894858, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.243.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:47:38.299784 10168 init.cc:377] Logged in from keytab as oryx/127.3.243.126@KRBTEST.COM (short username oryx)
I20260619 18:47:38.301152 10168 webserver.cc:460] Webserver started at http://127.3.243.126:45761/ using document root <none> and password file <none>
I20260619 18:47:38.301769 10168 fs_manager.cc:359] Metadata directory not provided
I20260619 18:47:38.301875 10168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:47:38.302140 10168 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:47:38.304920 10168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/master-0/data/instance:
uuid: "3e612a9d44274aadb7de143295bf9e38"
format_stamp: "Formatted at 2026-06-19 18:47:38 on dist-test-slave-m9wg"
server_key: "8683d6687b5c6f8ee7c3f694a99f50b3"
server_key_iv: "c2a119fa91aa48c8dddd2142f419fc99"
server_key_version: "encryptionkey@0"
I20260619 18:47:38.305613 10168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/master-0/wal/instance:
uuid: "3e612a9d44274aadb7de143295bf9e38"
format_stamp: "Formatted at 2026-06-19 18:47:38 on dist-test-slave-m9wg"
server_key: "8683d6687b5c6f8ee7c3f694a99f50b3"
server_key_iv: "c2a119fa91aa48c8dddd2142f419fc99"
server_key_version: "encryptionkey@0"
I20260619 18:47:38.310243 10168 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260619 18:47:38.313309 10183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:38.314465 10168 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 18:47:38.314606 10168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/master-0/wal
uuid: "3e612a9d44274aadb7de143295bf9e38"
format_stamp: "Formatted at 2026-06-19 18:47:38 on dist-test-slave-m9wg"
server_key: "8683d6687b5c6f8ee7c3f694a99f50b3"
server_key_iv: "c2a119fa91aa48c8dddd2142f419fc99"
server_key_version: "encryptionkey@0"
I20260619 18:47:38.314769 10168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:47:38.340901 10168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:47:38.345088 10168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:47:38.345363 10168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:47:38.362053 10168 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.126:41477
I20260619 18:47:38.362119 10234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.126:41477 every 8 connection(s)
I20260619 18:47:38.363454 10168 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
I20260619 18:47:38.367038 10235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:47:38.370157 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 10168
I20260619 18:47:38.370321 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/master-0/wal/instance
I20260619 18:47:38.370939 4045 external_mini_cluster.cc:1428] Setting key aca9fc42517645a4cde9dcbe83b57a99
I20260619 18:47:38.379416 10235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e612a9d44274aadb7de143295bf9e38: Bootstrap starting.
I20260619 18:47:38.382771 10235 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e612a9d44274aadb7de143295bf9e38: Neither blocks nor log segments found. Creating new log.
I20260619 18:47:38.383674 10235 log.cc:824] T 00000000000000000000000000000000 P 3e612a9d44274aadb7de143295bf9e38: Log is configured to *not* fsync() on all Append() calls
Jun 19 18:47:38 dist-test-slave-m9wg krb5kdc[10155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894857, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM
I20260619 18:47:38.386510 10235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e612a9d44274aadb7de143295bf9e38: No bootstrap required, opened a new log
I20260619 18:47:38.392431 10238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:38.372828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48934 (local address 127.3.243.126:41477)
0619 18:47:38.373563 (+ 735us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:38.373578 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:38.373626 (+ 48us) server_negotiation.cc:407] Connection header received
0619 18:47:38.375381 (+ 1755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:38.375410 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:38.375922 (+ 512us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:38.376394 (+ 472us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:38.377597 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:38.379479 (+ 1882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:38.380711 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:38.381709 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:38.381874 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:38.385644 (+ 3770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:38.385651 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:38.385664 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:38.385696 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:38.388577 (+ 2881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:38.389293 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:38.389300 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:38.389305 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:38.389474 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:38.389848 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:38.389852 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:38.389854 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:38.390329 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:38.390559 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:38.391041 (+ 482us) server_negotiation.cc:299] Negotiation successful
0619 18:47:38.391410 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":298,"threads_started":1}
I20260619 18:47:38.395081 10235 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e612a9d44274aadb7de143295bf9e38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e612a9d44274aadb7de143295bf9e38" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 41477 } }
I20260619 18:47:38.395519 10235 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e612a9d44274aadb7de143295bf9e38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:47:38.395605 10235 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e612a9d44274aadb7de143295bf9e38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e612a9d44274aadb7de143295bf9e38, State: Initialized, Role: FOLLOWER
I20260619 18:47:38.396083 10235 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e612a9d44274aadb7de143295bf9e38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e612a9d44274aadb7de143295bf9e38" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 41477 } }
I20260619 18:47:38.396323 10235 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3e612a9d44274aadb7de143295bf9e38 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:47:38.396420 10235 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e612a9d44274aadb7de143295bf9e38 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:47:38.396538 10235 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e612a9d44274aadb7de143295bf9e38 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:47:38.398878 10235 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e612a9d44274aadb7de143295bf9e38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e612a9d44274aadb7de143295bf9e38" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 41477 } }
I20260619 18:47:38.399389 10235 leader_election.cc:302] T 00000000000000000000000000000000 P 3e612a9d44274aadb7de143295bf9e38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e612a9d44274aadb7de143295bf9e38; no voters:
I20260619 18:47:38.399947 10235 leader_election.cc:288] T 00000000000000000000000000000000 P 3e612a9d44274aadb7de143295bf9e38 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:47:38.400067 10240 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e612a9d44274aadb7de143295bf9e38 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:47:38.400707 10240 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e612a9d44274aadb7de143295bf9e38 [term 1 LEADER]: Becoming Leader. State: Replica: 3e612a9d44274aadb7de143295bf9e38, State: Running, Role: LEADER
I20260619 18:47:38.401270 10240 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e612a9d44274aadb7de143295bf9e38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e612a9d44274aadb7de143295bf9e38" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 41477 } }
I20260619 18:47:38.401860 10235 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e612a9d44274aadb7de143295bf9e38 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:47:38.404116 10242 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e612a9d44274aadb7de143295bf9e38 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e612a9d44274aadb7de143295bf9e38. Latest consensus state: current_term: 1 leader_uuid: "3e612a9d44274aadb7de143295bf9e38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e612a9d44274aadb7de143295bf9e38" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 41477 } } }
I20260619 18:47:38.404999 10242 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e612a9d44274aadb7de143295bf9e38 [sys.catalog]: This master's current role is: LEADER
I20260619 18:47:38.404670 10241 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e612a9d44274aadb7de143295bf9e38 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3e612a9d44274aadb7de143295bf9e38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e612a9d44274aadb7de143295bf9e38" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 41477 } } }
I20260619 18:47:38.405481 10241 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e612a9d44274aadb7de143295bf9e38 [sys.catalog]: This master's current role is: LEADER
I20260619 18:47:38.406422 10249 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:47:38.410784 10249 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:47:38.418080 10249 catalog_manager.cc:1347] Generated new cluster ID: 301fc5ce1f33458494634dbfacabe160
I20260619 18:47:38.418186 10249 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:47:38.428620 10249 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:47:38.430294 10249 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:47:38.443538 10249 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3e612a9d44274aadb7de143295bf9e38: Generated new TSK 0
I20260619 18:47:38.444315 10249 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.3.243.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.243.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.243.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/krb5kdc/oryx_127.3.243.65.keytab.
Entry for principal oryx/127.3.243.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/krb5kdc/oryx_127.3.243.65.keytab.
I20260619 18:47:38.498194 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.65:0
--local_ip_for_outbound_sockets=127.3.243.65
--webserver_interface=127.3.243.65
--webserver_port=0
--tserver_master_addrs=127.3.243.126:41477
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/krb5kdc/oryx_127.3.243.65.keytab
--principal=oryx/127.3.243.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:37789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:47:38.806728 10261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:47:38.806988 10261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:47:38.807111 10261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:47:38.813931 10261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:47:38.814033 10261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:47:38.814186 10261 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.65
I20260619 18:47:38.823220 10261 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:37789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/krb5kdc/oryx_127.3.243.65.keytab
--principal=oryx/127.3.243.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.243.65
--webserver_port=0
--tserver_master_addrs=127.3.243.126:41477
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.65
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:47:38.824440 10261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:47:38.825917 10261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:47:38.837842 10267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:38.837885 10269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:38.837908 10266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:47:38.838624 10261 server_base.cc:1034] running on GCE node
I20260619 18:47:38.839112 10261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:47:38.839946 10261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:47:38.841150 10261 hybrid_clock.cc:648] HybridClock initialized: now 1781894858841084 us; error 40 us; skew 500 ppm
I20260619 18:47:38.841410 10261 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:47:38 dist-test-slave-m9wg krb5kdc[10155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894858, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.243.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:47:38.844260 10261 init.cc:377] Logged in from keytab as oryx/127.3.243.65@KRBTEST.COM (short username oryx)
I20260619 18:47:38.845499 10261 webserver.cc:460] Webserver started at http://127.3.243.65:44313/ using document root <none> and password file <none>
I20260619 18:47:38.846093 10261 fs_manager.cc:359] Metadata directory not provided
I20260619 18:47:38.846192 10261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:47:38.846422 10261 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:47:38.849108 10261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/ts-0/data/instance:
uuid: "69da4e79b53c43fc8becd20ad510c8b8"
format_stamp: "Formatted at 2026-06-19 18:47:38 on dist-test-slave-m9wg"
server_key: "1bd46fb8a572e51e756e0453120ab80a"
server_key_iv: "fa2989ee24f4672da138559764e9ac6e"
server_key_version: "encryptionkey@0"
I20260619 18:47:38.849771 10261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/ts-0/wal/instance:
uuid: "69da4e79b53c43fc8becd20ad510c8b8"
format_stamp: "Formatted at 2026-06-19 18:47:38 on dist-test-slave-m9wg"
server_key: "1bd46fb8a572e51e756e0453120ab80a"
server_key_iv: "fa2989ee24f4672da138559764e9ac6e"
server_key_version: "encryptionkey@0"
I20260619 18:47:38.854357 10261 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260619 18:47:38.857452 10276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:38.858651 10261 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 18:47:38.858783 10261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/ts-0/wal
uuid: "69da4e79b53c43fc8becd20ad510c8b8"
format_stamp: "Formatted at 2026-06-19 18:47:38 on dist-test-slave-m9wg"
server_key: "1bd46fb8a572e51e756e0453120ab80a"
server_key_iv: "fa2989ee24f4672da138559764e9ac6e"
server_key_version: "encryptionkey@0"
I20260619 18:47:38.858958 10261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:47:38.880419 10261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:47:38.883991 10261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:47:38.884263 10261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:47:38.885131 10261 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:47:38.886296 10261 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:47:38.886363 10261 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:38.886473 10261 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:47:38.886524 10261 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:38.914644 10261 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.65:34185
I20260619 18:47:38.914693 10388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.65:34185 every 8 connection(s)
I20260619 18:47:38.915799 10261 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/ts-0/data/info.pb
I20260619 18:47:38.918820 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 10261
I20260619 18:47:38.919016 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/ts-0/wal/instance
I20260619 18:47:38.919826 4045 external_mini_cluster.cc:1428] Setting key 31fe45928f58cf345f442e7938209220
Jun 19 18:47:38 dist-test-slave-m9wg krb5kdc[10155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894858, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.243.65@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM
I20260619 18:47:38.934979 10391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:38.918193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.65:54779 (local address 127.3.243.126:41477)
0619 18:47:38.918581 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:38.918587 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:38.919697 (+ 1110us) server_negotiation.cc:407] Connection header received
0619 18:47:38.921612 (+ 1915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:38.921626 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:38.921701 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:38.921812 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:38.923151 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:38.924107 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:38.925537 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:38.926085 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:38.926200 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:38.929766 (+ 3566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:38.929773 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:38.929777 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:38.929815 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:38.932191 (+ 2376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:38.932906 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:38.932910 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:38.932912 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:38.933017 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:38.933400 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:38.933403 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:38.933404 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:38.933654 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:38.933755 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:38.934487 (+ 732us) server_negotiation.cc:299] Negotiation successful
0619 18:47:38.934719 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":195,"threads_started":1}
I20260619 18:47:38.936262 10392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:38.918823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:41477 (local address 127.3.243.65:54779)
0619 18:47:38.919544 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:38.919585 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:38.921345 (+ 1760us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:38.921980 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:38.921987 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:38.922457 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:47:38.922986 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:38.923003 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:38.924241 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:38.924244 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:38.925411 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:38.925422 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:38.926250 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:38.926255 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:38.926435 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:38.927019 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:47:38.927039 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:47:38.929544 (+ 2505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:47:38.932306 (+ 2762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:38.932314 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:38.932328 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:38.932813 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:38.933104 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:38.933108 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:38.933111 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:38.933317 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:38.933760 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:47:38.933767 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:47:38.934140 (+ 373us) client_negotiation.cc:769] Sending connection context
0619 18:47:38.934431 (+ 291us) client_negotiation.cc:240] Negotiation successful
0619 18:47:38.934910 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":297,"threads_started":1}
I20260619 18:47:38.937626 10389 heartbeater.cc:347] Connected to a master server at 127.3.243.126:41477
I20260619 18:47:38.937949 10389 heartbeater.cc:464] Registering TS with master...
I20260619 18:47:38.938800 10389 heartbeater.cc:511] Master 127.3.243.126:41477 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.3.243.66@KRBTEST.COM; defaulting to no policy
I20260619 18:47:38.940881 10200 ts_manager.cc:194] Registered new tserver with Master: 69da4e79b53c43fc8becd20ad510c8b8 (127.3.243.65:34185)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.243.66@KRBTEST.COM" created.
I20260619 18:47:38.942778 10200 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.243.65@KRBTEST.COM'} at 127.3.243.65:54779
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.243.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/krb5kdc/oryx_127.3.243.66.keytab.
Entry for principal oryx/127.3.243.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/krb5kdc/oryx_127.3.243.66.keytab.
I20260619 18:47:38.962491 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.66:0
--local_ip_for_outbound_sockets=127.3.243.66
--webserver_interface=127.3.243.66
--webserver_port=0
--tserver_master_addrs=127.3.243.126:41477
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/krb5kdc/oryx_127.3.243.66.keytab
--principal=oryx/127.3.243.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:37789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:47:39.261639 10395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:47:39.261886 10395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:47:39.261981 10395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:47:39.269016 10395 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:47:39.269109 10395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:47:39.269225 10395 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.66
I20260619 18:47:39.278347 10395 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:37789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/krb5kdc/oryx_127.3.243.66.keytab
--principal=oryx/127.3.243.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.243.66
--webserver_port=0
--tserver_master_addrs=127.3.243.126:41477
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.66
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:47:39.279472 10395 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:47:39.280730 10395 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:47:39.293222 10401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:39.293190 10403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:39.293596 10400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:47:39.294191 10395 server_base.cc:1034] running on GCE node
I20260619 18:47:39.294754 10395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:47:39.295696 10395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:47:39.296890 10395 hybrid_clock.cc:648] HybridClock initialized: now 1781894859296838 us; error 48 us; skew 500 ppm
I20260619 18:47:39.297185 10395 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:47:39 dist-test-slave-m9wg krb5kdc[10155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894859, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.243.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:47:39.300086 10395 init.cc:377] Logged in from keytab as oryx/127.3.243.66@KRBTEST.COM (short username oryx)
I20260619 18:47:39.301394 10395 webserver.cc:460] Webserver started at http://127.3.243.66:38285/ using document root <none> and password file <none>
I20260619 18:47:39.302054 10395 fs_manager.cc:359] Metadata directory not provided
I20260619 18:47:39.302184 10395 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:47:39.302487 10395 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:47:39.305416 10395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/ts-1/data/instance:
uuid: "3f5657a7496c4e7b82d4244c01154242"
format_stamp: "Formatted at 2026-06-19 18:47:39 on dist-test-slave-m9wg"
server_key: "97b82f25e1c42724bbfe0d8b34007aec"
server_key_iv: "2b18c224b462f17668012ff6fd2848c0"
server_key_version: "encryptionkey@0"
I20260619 18:47:39.306133 10395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/ts-1/wal/instance:
uuid: "3f5657a7496c4e7b82d4244c01154242"
format_stamp: "Formatted at 2026-06-19 18:47:39 on dist-test-slave-m9wg"
server_key: "97b82f25e1c42724bbfe0d8b34007aec"
server_key_iv: "2b18c224b462f17668012ff6fd2848c0"
server_key_version: "encryptionkey@0"
I20260619 18:47:39.310787 10395 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260619 18:47:39.313958 10410 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:39.315124 10395 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 18:47:39.315244 10395 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/ts-1/wal
uuid: "3f5657a7496c4e7b82d4244c01154242"
format_stamp: "Formatted at 2026-06-19 18:47:39 on dist-test-slave-m9wg"
server_key: "97b82f25e1c42724bbfe0d8b34007aec"
server_key_iv: "2b18c224b462f17668012ff6fd2848c0"
server_key_version: "encryptionkey@0"
I20260619 18:47:39.315387 10395 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:47:39.336437 10395 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:47:39.340338 10395 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:47:39.340552 10395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:47:39.341365 10395 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:47:39.342540 10395 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:47:39.342608 10395 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:39.342701 10395 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:47:39.342742 10395 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:39.371978 10395 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.66:37147
I20260619 18:47:39.372011 10522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.66:37147 every 8 connection(s)
I20260619 18:47:39.373221 10395 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/ts-1/data/info.pb
I20260619 18:47:39.377210 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 10395
I20260619 18:47:39.377388 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/ts-1/wal/instance
I20260619 18:47:39.377889 4045 external_mini_cluster.cc:1428] Setting key bd92050fcbee0d0e91d427a11e2a50c6
Jun 19 18:47:39 dist-test-slave-m9wg krb5kdc[10155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894859, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.243.66@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM
I20260619 18:47:39.393911 10391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:39.375852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.66:51589 (local address 127.3.243.126:41477)
0619 18:47:39.376061 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:39.376067 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:39.377550 (+ 1483us) server_negotiation.cc:407] Connection header received
0619 18:47:39.379846 (+ 2296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:39.379852 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:39.379928 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:39.380030 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:39.381527 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:39.382462 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:39.383934 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:39.384541 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:39.384652 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:39.388139 (+ 3487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:39.388145 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:39.388169 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:39.388206 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:39.391125 (+ 2919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:39.391767 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:39.391773 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:39.391776 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:39.391924 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:39.392329 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:39.392333 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:39.392336 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:39.392639 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:39.392788 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:39.393382 (+ 594us) server_negotiation.cc:299] Negotiation successful
0619 18:47:39.393593 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 18:47:39.394712 10525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:39.376508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:41477 (local address 127.3.243.66:51589)
0619 18:47:39.377376 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:39.377446 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:39.379532 (+ 2086us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:39.380260 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:39.380268 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:39.380785 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:47:39.381345 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:39.381364 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:39.382579 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:39.382583 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:39.383801 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:39.383813 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:39.384735 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:39.384740 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:39.384924 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:39.385531 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:47:39.385552 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:47:39.387930 (+ 2378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:47:39.391265 (+ 3335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:39.391271 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:39.391283 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:39.391667 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:39.392015 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:39.392018 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:39.392021 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:39.392243 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:39.392746 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:47:39.392751 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:47:39.393035 (+ 284us) client_negotiation.cc:769] Sending connection context
0619 18:47:39.393250 (+ 215us) client_negotiation.cc:240] Negotiation successful
0619 18:47:39.393604 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":328,"threads_started":1}
I20260619 18:47:39.396142 10523 heartbeater.cc:347] Connected to a master server at 127.3.243.126:41477
I20260619 18:47:39.396461 10523 heartbeater.cc:464] Registering TS with master...
I20260619 18:47:39.397146 10523 heartbeater.cc:511] Master 127.3.243.126:41477 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.3.243.67@KRBTEST.COM; defaulting to no policy
I20260619 18:47:39.398521 10200 ts_manager.cc:194] Registered new tserver with Master: 3f5657a7496c4e7b82d4244c01154242 (127.3.243.66:37147)
I20260619 18:47:39.399358 10200 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.243.66@KRBTEST.COM'} at 127.3.243.66:51589
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.243.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.243.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/krb5kdc/oryx_127.3.243.67.keytab.
Entry for principal oryx/127.3.243.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/krb5kdc/oryx_127.3.243.67.keytab.
I20260619 18:47:39.421278 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.67:0
--local_ip_for_outbound_sockets=127.3.243.67
--webserver_interface=127.3.243.67
--webserver_port=0
--tserver_master_addrs=127.3.243.126:41477
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/krb5kdc/oryx_127.3.243.67.keytab
--principal=oryx/127.3.243.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:37789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:47:39.720026 10528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:47:39.720309 10528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:47:39.720438 10528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:47:39.727278 10528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:47:39.727380 10528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:47:39.727530 10528 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.67
I20260619 18:47:39.736639 10528 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:37789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/krb5kdc/oryx_127.3.243.67.keytab
--principal=oryx/127.3.243.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.243.67
--webserver_port=0
--tserver_master_addrs=127.3.243.126:41477
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.67
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:47:39.737778 10528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:47:39.738991 10528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:47:39.750674 10534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:39.750731 10533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:39.750770 10536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:47:39.751384 10528 server_base.cc:1034] running on GCE node
I20260619 18:47:39.751878 10528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:47:39.752696 10528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:47:39.753866 10528 hybrid_clock.cc:648] HybridClock initialized: now 1781894859753817 us; error 46 us; skew 500 ppm
I20260619 18:47:39.754127 10528 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:47:39 dist-test-slave-m9wg krb5kdc[10155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894859, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.243.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:47:39.756925 10528 init.cc:377] Logged in from keytab as oryx/127.3.243.67@KRBTEST.COM (short username oryx)
I20260619 18:47:39.758186 10528 webserver.cc:460] Webserver started at http://127.3.243.67:37007/ using document root <none> and password file <none>
I20260619 18:47:39.758989 10528 fs_manager.cc:359] Metadata directory not provided
I20260619 18:47:39.759119 10528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:47:39.759443 10528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:47:39.763295 10528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/ts-2/data/instance:
uuid: "7c75efa0e0da41d7883d6fc4bc7c6f89"
format_stamp: "Formatted at 2026-06-19 18:47:39 on dist-test-slave-m9wg"
server_key: "306d575bf5a85c02703afbb83a6c410a"
server_key_iv: "efbca80ac7b9fa5b44605e486e871f72"
server_key_version: "encryptionkey@0"
I20260619 18:47:39.764273 10528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/ts-2/wal/instance:
uuid: "7c75efa0e0da41d7883d6fc4bc7c6f89"
format_stamp: "Formatted at 2026-06-19 18:47:39 on dist-test-slave-m9wg"
server_key: "306d575bf5a85c02703afbb83a6c410a"
server_key_iv: "efbca80ac7b9fa5b44605e486e871f72"
server_key_version: "encryptionkey@0"
I20260619 18:47:39.769434 10528 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260619 18:47:39.772531 10543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:39.773653 10528 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260619 18:47:39.773790 10528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/ts-2/wal
uuid: "7c75efa0e0da41d7883d6fc4bc7c6f89"
format_stamp: "Formatted at 2026-06-19 18:47:39 on dist-test-slave-m9wg"
server_key: "306d575bf5a85c02703afbb83a6c410a"
server_key_iv: "efbca80ac7b9fa5b44605e486e871f72"
server_key_version: "encryptionkey@0"
I20260619 18:47:39.773949 10528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:47:39.789855 10528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:47:39.793574 10528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:47:39.793809 10528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:47:39.794610 10528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:47:39.795841 10528 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:47:39.795908 10528 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:39.796034 10528 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:47:39.796080 10528 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:39.824224 10528 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.67:36847
I20260619 18:47:39.824262 10655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.67:36847 every 8 connection(s)
I20260619 18:47:39.825421 10528 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/ts-2/data/info.pb
I20260619 18:47:39.826377 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 10528
I20260619 18:47:39.826527 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/ts-2/wal/instance
I20260619 18:47:39.826941 4045 external_mini_cluster.cc:1428] Setting key 1a477d71df8276285a10d19210466b20
Jun 19 18:47:39 dist-test-slave-m9wg krb5kdc[10155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894859, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.243.67@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM
I20260619 18:47:39.843300 10391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:39.827961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.67:59045 (local address 127.3.243.126:41477)
0619 18:47:39.828180 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:39.828193 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:39.829318 (+ 1125us) server_negotiation.cc:407] Connection header received
0619 18:47:39.831133 (+ 1815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:39.831138 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:39.831221 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:39.831303 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:39.832701 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:39.833422 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:39.834838 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:39.835237 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:39.835323 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:39.838361 (+ 3038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:39.838366 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:39.838369 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:39.838399 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:39.840798 (+ 2399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:39.841458 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:39.841463 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:39.841466 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:39.841603 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:39.842007 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:39.842010 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:39.842012 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:39.842268 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:39.842371 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:39.842903 (+ 532us) server_negotiation.cc:299] Negotiation successful
0619 18:47:39.843060 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 18:47:39.844208 10658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:39.828408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:41477 (local address 127.3.243.67:59045)
0619 18:47:39.829154 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:39.829213 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:39.830881 (+ 1668us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:39.831484 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:39.831491 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:39.831966 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:47:39.832549 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:39.832566 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:39.833533 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:39.833536 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:39.834724 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:39.834736 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:39.835393 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:39.835398 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:39.835570 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:39.836190 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:47:39.836225 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:47:39.838174 (+ 1949us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:47:39.840918 (+ 2744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:39.840926 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:39.840940 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:39.841351 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:39.841706 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:39.841709 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:39.841711 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:39.841913 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:39.842353 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:47:39.842358 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:47:39.842623 (+ 265us) client_negotiation.cc:769] Sending connection context
0619 18:47:39.842830 (+ 207us) client_negotiation.cc:240] Negotiation successful
0619 18:47:39.843165 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":298,"threads_started":1}
I20260619 18:47:39.845386 10656 heartbeater.cc:347] Connected to a master server at 127.3.243.126:41477
I20260619 18:47:39.845621 10656 heartbeater.cc:464] Registering TS with master...
I20260619 18:47:39.846267 10656 heartbeater.cc:511] Master 127.3.243.126:41477 requested a full tablet report, sending...
I20260619 18:47:39.847494 10200 ts_manager.cc:194] Registered new tserver with Master: 7c75efa0e0da41d7883d6fc4bc7c6f89 (127.3.243.67:36847)
I20260619 18:47:39.848269 10200 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.243.67@KRBTEST.COM'} at 127.3.243.67:59045
I20260619 18:47:39.853653 4045 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.3.243.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.243.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.243.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/krb5kdc/oryx_127.3.243.125.keytab.
Entry for principal oryx/127.3.243.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/krb5kdc/oryx_127.3.243.125.keytab.
I20260619 18:47:39.892962 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.125:37445
--webserver_interface=127.3.243.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/krb5kdc/oryx_127.3.243.125.keytab
--principal=oryx/127.3.243.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:37789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.126:41477,127.3.243.125:37445
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 18:47:39.945631 10389 heartbeater.cc:503] Master 127.3.243.126:41477 was elected leader, sending a full tablet report...
W20260619 18:47:40.195206 10661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:47:40.195492 10661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:47:40.195621 10661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:47:40.202905 10661 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:47:40.203017 10661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:47:40.203094 10661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:47:40.203174 10661 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:47:40.203238 10661 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:47:40.213009 10661 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:37789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.243.126:41477,127.3.243.125:37445
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/krb5kdc/oryx_127.3.243.125.keytab
--principal=oryx/127.3.243.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.125:37445
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.243.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
W20260619 18:47:40.213356 10661 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.3.243.126:41477,127.3.243.125:37445'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260619 18:47:40.214363 10661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:47:40.215600 10661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:47:40.226975 10666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:40.227054 10667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:40.227020 10669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:47:40.227886 10661 server_base.cc:1034] running on GCE node
I20260619 18:47:40.228633 10661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:47:40.229946 10661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:47:40.231166 10661 hybrid_clock.cc:648] HybridClock initialized: now 1781894860231103 us; error 32 us; skew 500 ppm
I20260619 18:47:40.231467 10661 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:47:40 dist-test-slave-m9wg krb5kdc[10155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894860, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.243.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:47:40.235129 10661 init.cc:377] Logged in from keytab as oryx/127.3.243.125@KRBTEST.COM (short username oryx)
I20260619 18:47:40.236567 10661 webserver.cc:460] Webserver started at http://127.3.243.125:44565/ using document root <none> and password file <none>
I20260619 18:47:40.237198 10661 fs_manager.cc:359] Metadata directory not provided
I20260619 18:47:40.237304 10661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:47:40.237571 10661 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:47:40.240392 10661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/master-1/data/instance:
uuid: "ae1e1640d5134c40a77de2ae1fd29530"
format_stamp: "Formatted at 2026-06-19 18:47:40 on dist-test-slave-m9wg"
server_key: "0552ce0f9b1072045186b3d922772003"
server_key_iv: "bd635aa232fb134d2ec5b1a82267b91b"
server_key_version: "encryptionkey@0"
I20260619 18:47:40.241096 10661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/master-1/wal/instance:
uuid: "ae1e1640d5134c40a77de2ae1fd29530"
format_stamp: "Formatted at 2026-06-19 18:47:40 on dist-test-slave-m9wg"
server_key: "0552ce0f9b1072045186b3d922772003"
server_key_iv: "bd635aa232fb134d2ec5b1a82267b91b"
server_key_version: "encryptionkey@0"
I20260619 18:47:40.245806 10661 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 18:47:40.248970 10676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:40.250201 10661 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260619 18:47:40.250334 10661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/master-1/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/master-1/wal
uuid: "ae1e1640d5134c40a77de2ae1fd29530"
format_stamp: "Formatted at 2026-06-19 18:47:40 on dist-test-slave-m9wg"
server_key: "0552ce0f9b1072045186b3d922772003"
server_key_iv: "bd635aa232fb134d2ec5b1a82267b91b"
server_key_version: "encryptionkey@0"
I20260619 18:47:40.250499 10661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:47:40.286356 10661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:47:40.290596 10661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:47:40.290817 10661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:47:40.307441 10661 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.125:37445
I20260619 18:47:40.307474 10727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.125:37445 every 8 connection(s)
I20260619 18:47:40.308768 10661 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/master-1/data/info.pb
I20260619 18:47:40.312278 10728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:47:40.318660 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 10661
I20260619 18:47:40.318634 10728 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.243.126" port: 41477 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 18:47:40 dist-test-slave-m9wg krb5kdc[10155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894860, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.243.125@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM
Jun 19 18:47:40 dist-test-slave-m9wg krb5kdc[10155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894857, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.243.125@KRBTEST.COM
I20260619 18:47:40.337141 10391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:40.321429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32960 (local address 127.3.243.126:41477)
0619 18:47:40.321613 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:40.321619 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:40.322374 (+ 755us) server_negotiation.cc:407] Connection header received
0619 18:47:40.323328 (+ 954us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:40.323334 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:40.323406 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:40.323501 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:40.325070 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:40.326167 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:40.327749 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:40.328134 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:40.328273 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:40.331533 (+ 3260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:40.331538 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:40.331542 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:40.331572 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:40.334179 (+ 2607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:40.334911 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:40.334917 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:40.334920 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:40.335046 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:40.335419 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:40.335425 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:40.335428 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:40.335721 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:40.335849 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:40.336619 (+ 770us) server_negotiation.cc:299] Negotiation successful
0619 18:47:40.336824 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 18:47:40.338336 10731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:40.320662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45648 (local address 127.3.243.125:37445)
0619 18:47:40.321356 (+ 694us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:40.321365 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:40.321399 (+ 34us) server_negotiation.cc:407] Connection header received
0619 18:47:40.323078 (+ 1679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:40.323097 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:40.323346 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:40.323548 (+ 202us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:40.325228 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:40.326730 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:40.327916 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:40.328450 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:40.328552 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:40.332493 (+ 3941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:40.332500 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:40.332510 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:40.332540 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:40.335093 (+ 2553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:40.335686 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:40.335692 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:40.335694 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:40.335805 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:40.336196 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:40.336201 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:40.336204 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:40.336601 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:40.336876 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:40.337204 (+ 328us) server_negotiation.cc:299] Negotiation successful
0619 18:47:40.337420 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":513,"thread_start_us":319,"threads_started":1}
I20260619 18:47:40.338316 10732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:40.321429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:41477 (local address 127.0.0.1:32960)
0619 18:47:40.322221 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:40.322265 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:40.323085 (+ 820us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:40.323572 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:40.323578 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:40.324407 (+ 829us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:47:40.324907 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:40.324932 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:40.326336 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:40.326340 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:40.327605 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:40.327616 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:40.328388 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:40.328396 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:40.328547 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:40.329173 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:47:40.329211 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:47:40.331358 (+ 2147us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:47:40.334299 (+ 2941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:40.334309 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:40.334320 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:40.334807 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:40.335143 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:40.335148 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:40.335152 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:40.335318 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:40.335840 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:47:40.335847 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:47:40.336179 (+ 332us) client_negotiation.cc:769] Sending connection context
0619 18:47:40.336544 (+ 365us) client_negotiation.cc:240] Negotiation successful
0619 18:47:40.337006 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"mutex_wait_us":3,"thread_start_us":372,"threads_started":1}
I20260619 18:47:40.340265 10728 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.243.125" port: 37445 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 18:47:40 dist-test-slave-m9wg krb5kdc[10155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894860, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.243.125@KRBTEST.COM for oryx/127.3.243.125@KRBTEST.COM
I20260619 18:47:40.350952 10732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:40.341066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.125:37445 (local address 127.0.0.1:45654)
0619 18:47:40.341215 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:40.341232 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:40.341310 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:40.341514 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:40.341517 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:40.341805 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:47:40.341925 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:40.341932 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:40.342824 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:40.342827 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:40.343633 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:40.343642 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:40.344285 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:40.344289 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:40.344427 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:40.344865 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:47:40.344880 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:47:40.346637 (+ 1757us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:47:40.349162 (+ 2525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:40.349165 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:40.349168 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:40.349496 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:40.349777 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:40.349780 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:40.349782 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:40.349901 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:40.350322 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:47:40.350325 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:47:40.350426 (+ 101us) client_negotiation.cc:769] Sending connection context
0619 18:47:40.350523 (+ 97us) client_negotiation.cc:240] Negotiation successful
0619 18:47:40.350719 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260619 18:47:40.351115 10731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:40.341125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45654 (local address 127.3.243.125:37445)
0619 18:47:40.341274 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:40.341278 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:40.341291 (+ 13us) server_negotiation.cc:407] Connection header received
0619 18:47:40.341375 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:40.341379 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:40.341426 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:40.341493 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:40.342026 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:40.342709 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:40.343750 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:40.344108 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:40.344270 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:40.346788 (+ 2518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:40.346792 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:40.346795 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:40.346823 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:40.349049 (+ 2226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:40.349581 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:40.349585 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:40.349587 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:40.349695 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:40.349978 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:40.349981 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:40.349983 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:40.350238 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:40.350336 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:40.350624 (+ 288us) server_negotiation.cc:299] Negotiation successful
0619 18:47:40.350851 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":57}
I20260619 18:47:40.355541 10728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae1e1640d5134c40a77de2ae1fd29530: Bootstrap starting.
I20260619 18:47:40.358546 10728 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae1e1640d5134c40a77de2ae1fd29530: Neither blocks nor log segments found. Creating new log.
I20260619 18:47:40.359524 10728 log.cc:824] T 00000000000000000000000000000000 P ae1e1640d5134c40a77de2ae1fd29530: Log is configured to *not* fsync() on all Append() calls
I20260619 18:47:40.362258 10728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae1e1640d5134c40a77de2ae1fd29530: No bootstrap required, opened a new log
I20260619 18:47:40.372695 10728 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae1e1640d5134c40a77de2ae1fd29530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e612a9d44274aadb7de143295bf9e38" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 41477 } } peers { permanent_uuid: "ae1e1640d5134c40a77de2ae1fd29530" member_type: VOTER last_known_addr { host: "127.3.243.125" port: 37445 } }
I20260619 18:47:40.373085 10728 raft_consensus.cc:374] T 00000000000000000000000000000000 P ae1e1640d5134c40a77de2ae1fd29530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:47:40.373173 10728 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae1e1640d5134c40a77de2ae1fd29530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae1e1640d5134c40a77de2ae1fd29530, State: Initialized, Role: FOLLOWER
I20260619 18:47:40.373821 10728 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae1e1640d5134c40a77de2ae1fd29530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e612a9d44274aadb7de143295bf9e38" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 41477 } } peers { permanent_uuid: "ae1e1640d5134c40a77de2ae1fd29530" member_type: VOTER last_known_addr { host: "127.3.243.125" port: 37445 } }
I20260619 18:47:40.374758 10734 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae1e1640d5134c40a77de2ae1fd29530 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e612a9d44274aadb7de143295bf9e38" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 41477 } } peers { permanent_uuid: "ae1e1640d5134c40a77de2ae1fd29530" member_type: VOTER last_known_addr { host: "127.3.243.125" port: 37445 } } }
I20260619 18:47:40.375128 10734 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae1e1640d5134c40a77de2ae1fd29530 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:47:40.375703 10728 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae1e1640d5134c40a77de2ae1fd29530 [sys.catalog]: configured and running, proceeding with master startup.
W20260619 18:47:40.383764 10745 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ae1e1640d5134c40a77de2ae1fd29530: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 18:47:40.383893 10745 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 18:47:40.385674 10661 master_runner.cc:428] Detected that this master ae1e1640d5134c40a77de2ae1fd29530 is joining an existing cluster
I20260619 18:47:40.385754 10661 master_runner.cc:432] Initiating AddMaster RPC to add 127.3.243.125:37445
I20260619 18:47:40.401925 10523 heartbeater.cc:503] Master 127.3.243.126:41477 was elected leader, sending a full tablet report...
I20260619 18:47:40.404954 10391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:40.389206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32976 (local address 127.3.243.126:41477)
0619 18:47:40.389381 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:40.389388 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:40.390048 (+ 660us) server_negotiation.cc:407] Connection header received
0619 18:47:40.390350 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:40.390356 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:40.390437 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:40.390528 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:40.392047 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:40.393156 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:40.395188 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:40.395805 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:40.395899 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:40.398441 (+ 2542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:40.398448 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:40.398453 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:40.398493 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:40.401799 (+ 3306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:40.402565 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:40.402571 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:40.402575 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:40.402743 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:40.403173 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:40.403192 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:40.403195 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:40.403565 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:40.403686 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:40.404348 (+ 662us) server_negotiation.cc:299] Negotiation successful
0619 18:47:40.404592 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 18:47:40.405028 10751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:40.389281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:41477 (local address 127.0.0.1:32976)
0619 18:47:40.389943 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:40.389962 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:40.390169 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:40.390691 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:40.390696 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:40.391488 (+ 792us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:47:40.391865 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:40.391878 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:40.393381 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:40.393386 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:40.395007 (+ 1621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:40.395021 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:40.396217 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:40.396225 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:40.396401 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:40.396968 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:47:40.396990 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:47:40.398236 (+ 1246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:47:40.401947 (+ 3711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:40.401952 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:40.401956 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:40.402423 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:40.402885 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:40.402890 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:40.402893 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:40.403048 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:40.403785 (+ 737us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:47:40.403790 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:47:40.404010 (+ 220us) client_negotiation.cc:769] Sending connection context
0619 18:47:40.404214 (+ 204us) client_negotiation.cc:240] Negotiation successful
0619 18:47:40.404521 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"mutex_wait_us":3,"thread_start_us":369,"threads_started":1}
I20260619 18:47:40.405874 10731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:40.389244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45660 (local address 127.3.243.125:37445)
0619 18:47:40.389641 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:40.389646 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:40.390380 (+ 734us) server_negotiation.cc:407] Connection header received
0619 18:47:40.390503 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:40.390507 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:40.390557 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:40.390626 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:40.391882 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:40.392917 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:40.394756 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:40.395441 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:40.395529 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:40.397924 (+ 2395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:40.397933 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:40.397937 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:40.397974 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:40.401184 (+ 3210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:40.401965 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:40.401970 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:40.401973 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:40.402253 (+ 280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:40.403787 (+ 1534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:40.403792 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:40.403795 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:40.404095 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:40.404250 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:40.404781 (+ 531us) server_negotiation.cc:299] Negotiation successful
0619 18:47:40.404992 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267}
I20260619 18:47:40.406737 10752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:40.389590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.125:37445 (local address 127.0.0.1:45660)
0619 18:47:40.390300 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:40.390315 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:40.390416 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:40.390661 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:40.390666 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:40.391272 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:47:40.391731 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:40.391744 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:40.393125 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:40.393129 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:40.394601 (+ 1472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:40.394616 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:40.395842 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:40.395848 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:40.396027 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:40.396623 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:47:40.396648 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:47:40.397756 (+ 1108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:47:40.401309 (+ 3553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:40.401313 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:40.401316 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:40.401851 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:40.402383 (+ 532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:40.402387 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:40.402390 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:40.402625 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:40.404337 (+ 1712us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:47:40.404341 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:47:40.404517 (+ 176us) client_negotiation.cc:769] Sending connection context
0619 18:47:40.406022 (+ 1505us) client_negotiation.cc:240] Negotiation successful
0619 18:47:40.406280 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"mutex_wait_us":22,"spinlock_wait_cycles":26496,"thread_start_us":419,"threads_started":1}
Jun 19 18:47:40 dist-test-slave-m9wg krb5kdc[10155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894858, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.243.126@KRBTEST.COM for oryx/127.3.243.125@KRBTEST.COM
I20260619 18:47:40.425137 10731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:40.411264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45662 (local address 127.3.243.125:37445)
0619 18:47:40.411427 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:40.411432 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:40.412025 (+ 593us) server_negotiation.cc:407] Connection header received
0619 18:47:40.412344 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:40.412349 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:40.412438 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:40.412522 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:40.413515 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:40.414251 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:40.415505 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:40.415954 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:40.416069 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:40.419404 (+ 3335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:40.419410 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:40.419415 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:40.419444 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:40.422580 (+ 3136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:40.423215 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:40.423221 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:40.423224 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:40.423350 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:40.423687 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:40.423692 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:40.423695 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:40.423939 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:40.424067 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:40.424539 (+ 472us) server_negotiation.cc:299] Negotiation successful
0619 18:47:40.424762 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 18:47:40.425180 10754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:40.411249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.125:37445 (local address 127.0.0.1:45662)
0619 18:47:40.411897 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:40.411931 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:40.412091 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:40.412580 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:40.412587 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:40.413176 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:47:40.413364 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:40.413376 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:40.414368 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:40.414373 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:40.415357 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:40.415369 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:40.416108 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:40.416114 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:40.416331 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:40.416860 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:47:40.416876 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:47:40.419174 (+ 2298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:47:40.422706 (+ 3532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:40.422718 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:40.422725 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:40.423098 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:40.423448 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:40.423454 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:40.423457 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:40.423589 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:40.424062 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:47:40.424070 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:47:40.424280 (+ 210us) client_negotiation.cc:769] Sending connection context
0619 18:47:40.424439 (+ 159us) client_negotiation.cc:240] Negotiation successful
0619 18:47:40.424762 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":286,"threads_started":1}
I20260619 18:47:40.427367 10200 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.3.243.125:37445: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "ae1e1640d5134c40a77de2ae1fd29530"
member_type: NON_VOTER
last_known_addr {
host: "127.3.243.125"
port: 37445
}
attrs {
promote: true
}
}
dest_uuid: "3e612a9d44274aadb7de143295bf9e38"
cas_config_opid_index: -1
I20260619 18:47:40.428372 10200 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e612a9d44274aadb7de143295bf9e38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3e612a9d44274aadb7de143295bf9e38" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 41477 } } peers { permanent_uuid: "ae1e1640d5134c40a77de2ae1fd29530" member_type: NON_VOTER last_known_addr { host: "127.3.243.125" port: 37445 } attrs { promote: true } }
WARNING: no policy specified for oryx/127.3.243.124@KRBTEST.COM; defaulting to no policy
I20260619 18:47:40.431278 10757 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3e612a9d44274aadb7de143295bf9e38 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER ae1e1640d5134c40a77de2ae1fd29530 (127.3.243.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3e612a9d44274aadb7de143295bf9e38" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 41477 } } peers { permanent_uuid: "ae1e1640d5134c40a77de2ae1fd29530" member_type: NON_VOTER last_known_addr { host: "127.3.243.125" port: 37445 } attrs { promote: true } } }
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.243.124@KRBTEST.COM" created.
I20260619 18:47:40.432786 10703 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae1e1640d5134c40a77de2ae1fd29530 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:47:40.434482 10757 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.3.243.125:37445
I20260619 18:47:40.434895 10757 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e612a9d44274aadb7de143295bf9e38 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "3e612a9d44274aadb7de143295bf9e38" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3e612a9d44274aadb7de143295bf9e38" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 41477 } } peers { permanent_uuid: "ae1e1640d5134c40a77de2ae1fd29530" member_type: NON_VOTER last_known_addr { host: "127.3.243.125" port: 37445 } attrs { promote: true } } }
I20260619 18:47:40.435237 10661 master.cc:537] Master@127.3.243.125:37445 shutting down...
I20260619 18:47:40.435091 10757 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e612a9d44274aadb7de143295bf9e38 [sys.catalog]: This master's current role is: LEADER
I20260619 18:47:40.442880 10703 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ae1e1640d5134c40a77de2ae1fd29530 [term 1 FOLLOWER]: Refusing update from remote peer 3e612a9d44274aadb7de143295bf9e38: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260619 18:47:40.444316 10756 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3e612a9d44274aadb7de143295bf9e38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae1e1640d5134c40a77de2ae1fd29530" member_type: NON_VOTER last_known_addr { host: "127.3.243.125" port: 37445 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 18:47:40.447530 10187 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260619 18:47:40.450714 10661 raft_consensus.cc:2229] T 00000000000000000000000000000000 P ae1e1640d5134c40a77de2ae1fd29530 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:47:40.451299 10661 raft_consensus.cc:2258] T 00000000000000000000000000000000 P ae1e1640d5134c40a77de2ae1fd29530 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:47:40.451563 10661 tablet_replica.cc:330] T 00000000000000000000000000000000 P ae1e1640d5134c40a77de2ae1fd29530: stopping tablet replica
W20260619 18:47:40.453547 10187 consensus_peers.cc:487] T 00000000000000000000000000000000 P 3e612a9d44274aadb7de143295bf9e38 -> Peer ae1e1640d5134c40a77de2ae1fd29530 (127.3.243.125:37445): Couldn't send request to peer ae1e1640d5134c40a77de2ae1fd29530. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260619 18:47:40.468182 10661 master.cc:552] Master@127.3.243.125:37445 shutdown complete.
I20260619 18:47:40.468286 10661 master_runner.cc:305] Clearing existing system tablet
I20260619 18:47:40.468659 10661 fs_manager.cc:359] Metadata directory not provided
I20260619 18:47:40.468770 10661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.243.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/krb5kdc/oryx_127.3.243.124.keytab.
Entry for principal oryx/127.3.243.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/krb5kdc/oryx_127.3.243.124.keytab.
I20260619 18:47:40.471196 10661 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260619 18:47:40.471378 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.124:32923
--webserver_interface=127.3.243.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/krb5kdc/oryx_127.3.243.124.keytab
--principal=oryx/127.3.243.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:37789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.126:41477,127.3.243.125:37445,127.3.243.124:32923
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 18:47:40.472666 10762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:40.472941 10661 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:47:40.473193 10661 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 18:47:40.473278 10661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/master-1/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/master-1/wal
uuid: "ae1e1640d5134c40a77de2ae1fd29530"
format_stamp: "Formatted at 2026-06-19 18:47:40 on dist-test-slave-m9wg"
server_key: "0552ce0f9b1072045186b3d922772003"
server_key_iv: "bd635aa232fb134d2ec5b1a82267b91b"
server_key_version: "encryptionkey@0"
I20260619 18:47:40.476786 10661 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P ae1e1640d5134c40a77de2ae1fd29530: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:47:40.482203 10661 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P ae1e1640d5134c40a77de2ae1fd29530: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 18:47:40.482350 10661 log.cc:1196] T 00000000000000000000000000000000 P ae1e1640d5134c40a77de2ae1fd29530: Deleting WAL directory at /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260619 18:47:40.482717 10661 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P ae1e1640d5134c40a77de2ae1fd29530: Deleting consensus metadata
I20260619 18:47:40.482951 10661 master_runner.cc:315] Copying system tablet from 127.3.243.126:41477
I20260619 18:47:40.496593 10661 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P ae1e1640d5134c40a77de2ae1fd29530: tablet copy: Beginning tablet copy session from remote peer at address 127.3.243.126:41477
I20260619 18:47:40.512082 10391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:40.497541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32988 (local address 127.3.243.126:41477)
0619 18:47:40.497748 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:40.497754 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:40.498273 (+ 519us) server_negotiation.cc:407] Connection header received
0619 18:47:40.498469 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:40.498476 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:40.498573 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:40.498686 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:40.499773 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:40.500909 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:40.502817 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:40.503488 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:40.503613 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:40.505849 (+ 2236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:40.505856 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:40.505862 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:40.505902 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:40.509330 (+ 3428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:40.510082 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:40.510088 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:40.510092 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:40.510244 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:40.510619 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:40.510624 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:40.510627 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:40.510979 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:40.511129 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:40.511527 (+ 398us) server_negotiation.cc:299] Negotiation successful
0619 18:47:40.511751 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 18:47:40.512125 10769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:40.497472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:41477 (local address 127.0.0.1:32988)
0619 18:47:40.498134 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:40.498157 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:40.498333 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:40.498743 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:40.498747 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:40.499304 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:47:40.499601 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:40.499614 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:40.501073 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:40.501079 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:40.502645 (+ 1566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:40.502663 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:40.503637 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:40.503643 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:40.503837 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:40.504544 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:47:40.504571 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:47:40.505644 (+ 1073us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:47:40.509460 (+ 3816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:40.509465 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:40.509470 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:40.509968 (+ 498us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:40.510353 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:40.510358 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:40.510362 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:40.510518 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:40.511094 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:47:40.511098 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:47:40.511280 (+ 182us) client_negotiation.cc:769] Sending connection context
0619 18:47:40.511425 (+ 145us) client_negotiation.cc:240] Negotiation successful
0619 18:47:40.511718 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":389,"threads_started":1}
I20260619 18:47:40.513275 10220 tablet_copy_service.cc:140] P 3e612a9d44274aadb7de143295bf9e38: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer ae1e1640d5134c40a77de2ae1fd29530 ({username='oryx', principal='oryx/127.3.243.125@KRBTEST.COM'} at 127.0.0.1:32988)
I20260619 18:47:40.513597 10220 tablet_copy_service.cc:161] P 3e612a9d44274aadb7de143295bf9e38: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer ae1e1640d5134c40a77de2ae1fd29530 at {username='oryx', principal='oryx/127.3.243.125@KRBTEST.COM'} at 127.0.0.1:32988: session id = ae1e1640d5134c40a77de2ae1fd29530-00000000000000000000000000000000
I20260619 18:47:40.518220 10220 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 3e612a9d44274aadb7de143295bf9e38: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 18:47:40.520543 10661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 18:47:40.527323 10661 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P ae1e1640d5134c40a77de2ae1fd29530: tablet copy: Starting download of 0 data blocks...
I20260619 18:47:40.527688 10661 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P ae1e1640d5134c40a77de2ae1fd29530: tablet copy: Starting download of 1 WAL segments...
I20260619 18:47:40.530707 10661 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P ae1e1640d5134c40a77de2ae1fd29530: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 18:47:40.535344 10220 tablet_copy_service.cc:342] P 3e612a9d44274aadb7de143295bf9e38: Request end of tablet copy session ae1e1640d5134c40a77de2ae1fd29530-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.3.243.125@KRBTEST.COM'} at 127.0.0.1:32988
I20260619 18:47:40.535604 10220 tablet_copy_service.cc:433] P 3e612a9d44274aadb7de143295bf9e38: ending tablet copy session ae1e1640d5134c40a77de2ae1fd29530-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer ae1e1640d5134c40a77de2ae1fd29530
I20260619 18:47:40.547430 10661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:47:40.549957 10770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:40.551033 10771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:40.557840 10773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:47:40.560473 10661 server_base.cc:1034] running on GCE node
I20260619 18:47:40.560882 10661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:47:40.561661 10661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:47:40.562897 10661 hybrid_clock.cc:648] HybridClock initialized: now 1781894860562814 us; error 55 us; skew 500 ppm
I20260619 18:47:40.563159 10661 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:47:40 dist-test-slave-m9wg krb5kdc[10155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894860, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.243.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:47:40.565502 10661 init.cc:377] Logged in from keytab as oryx/127.3.243.125@KRBTEST.COM (short username oryx)
I20260619 18:47:40.569176 10661 webserver.cc:460] Webserver started at http://127.3.243.125:34103/ using document root <none> and password file <none>
I20260619 18:47:40.569555 10661 fs_manager.cc:359] Metadata directory not provided
I20260619 18:47:40.569640 10661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:47:40.572726 10661 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 18:47:40.574685 10780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:40.575107 10661 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260619 18:47:40.575255 10661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/master-1/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/master-1/wal
uuid: "ae1e1640d5134c40a77de2ae1fd29530"
format_stamp: "Formatted at 2026-06-19 18:47:40 on dist-test-slave-m9wg"
server_key: "0552ce0f9b1072045186b3d922772003"
server_key_iv: "bd635aa232fb134d2ec5b1a82267b91b"
server_key_version: "encryptionkey@0"
I20260619 18:47:40.575413 10661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:47:40.610929 10661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:47:40.614861 10661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:47:40.660552 10661 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.125:37445
I20260619 18:47:40.660607 10831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.125:37445 every 8 connection(s)
I20260619 18:47:40.661792 10661 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/master-1/data/info.pb
I20260619 18:47:40.668334 10832 sys_catalog.cc:263] Verifying existing consensus state
I20260619 18:47:40.672201 10832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae1e1640d5134c40a77de2ae1fd29530: Bootstrap starting.
I20260619 18:47:40.689946 10832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae1e1640d5134c40a77de2ae1fd29530: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 18:47:40.690464 10832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae1e1640d5134c40a77de2ae1fd29530: Bootstrap complete.
I20260619 18:47:40.691527 10832 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae1e1640d5134c40a77de2ae1fd29530 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3e612a9d44274aadb7de143295bf9e38" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 41477 } } peers { permanent_uuid: "ae1e1640d5134c40a77de2ae1fd29530" member_type: NON_VOTER last_known_addr { host: "127.3.243.125" port: 37445 } attrs { promote: true } }
I20260619 18:47:40.691838 10832 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae1e1640d5134c40a77de2ae1fd29530 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ae1e1640d5134c40a77de2ae1fd29530, State: Initialized, Role: LEARNER
I20260619 18:47:40.692122 10832 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae1e1640d5134c40a77de2ae1fd29530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3e612a9d44274aadb7de143295bf9e38" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 41477 } } peers { permanent_uuid: "ae1e1640d5134c40a77de2ae1fd29530" member_type: NON_VOTER last_known_addr { host: "127.3.243.125" port: 37445 } attrs { promote: true } }
I20260619 18:47:40.693043 10835 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae1e1640d5134c40a77de2ae1fd29530 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3e612a9d44274aadb7de143295bf9e38" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 41477 } } peers { permanent_uuid: "ae1e1640d5134c40a77de2ae1fd29530" member_type: NON_VOTER last_known_addr { host: "127.3.243.125" port: 37445 } attrs { promote: true } } }
I20260619 18:47:40.693382 10832 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae1e1640d5134c40a77de2ae1fd29530 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:47:40.693449 10835 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae1e1640d5134c40a77de2ae1fd29530 [sys.catalog]: This master's current role is: LEARNER
I20260619 18:47:40.704089 10846 catalog_manager.cc:1259] Loaded cluster ID: 301fc5ce1f33458494634dbfacabe160
I20260619 18:47:40.704270 10846 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ae1e1640d5134c40a77de2ae1fd29530: loading cluster ID for follower catalog manager: success
I20260619 18:47:40.707942 10846 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ae1e1640d5134c40a77de2ae1fd29530: acquiring CA information for follower catalog manager: success
I20260619 18:47:40.710479 10846 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ae1e1640d5134c40a77de2ae1fd29530: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260619 18:47:40.840796 10763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:47:40.841105 10763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:47:40.841251 10763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:47:40.848379 10763 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:47:40.848507 10763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:47:40.848572 10763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:47:40.848629 10763 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:47:40.848670 10763 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:47:40.850458 10656 heartbeater.cc:503] Master 127.3.243.126:41477 was elected leader, sending a full tablet report...
I20260619 18:47:40.858409 10763 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:37789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.243.126:41477,127.3.243.125:37445,127.3.243.124:32923
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/krb5kdc/oryx_127.3.243.124.keytab
--principal=oryx/127.3.243.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.124:32923
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.243.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:47:40.859655 10763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:47:40.860965 10763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:47:40.871907 10852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:40.871958 10851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:40.871977 10854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:47:40.872726 10763 server_base.cc:1034] running on GCE node
I20260619 18:47:40.873407 10763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:47:40.874707 10763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:47:40.875885 10763 hybrid_clock.cc:648] HybridClock initialized: now 1781894860875836 us; error 35 us; skew 500 ppm
I20260619 18:47:40.876271 10763 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:47:40 dist-test-slave-m9wg krb5kdc[10155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894860, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.243.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:47:40.879170 10763 init.cc:377] Logged in from keytab as oryx/127.3.243.124@KRBTEST.COM (short username oryx)
I20260619 18:47:40.880561 10763 webserver.cc:460] Webserver started at http://127.3.243.124:40191/ using document root <none> and password file <none>
I20260619 18:47:40.881188 10763 fs_manager.cc:359] Metadata directory not provided
I20260619 18:47:40.881310 10763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:47:40.881582 10763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:47:40.884528 10763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/master-2/data/instance:
uuid: "ed237dce23ef407fa74b615f8f6c914f"
format_stamp: "Formatted at 2026-06-19 18:47:40 on dist-test-slave-m9wg"
server_key: "8c344ee2d928dc2fc5e5819090cf426f"
server_key_iv: "4d14c19b4b0fad8aa5a8b2e3623d31f1"
server_key_version: "encryptionkey@0"
I20260619 18:47:40.885303 10763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/master-2/wal/instance:
uuid: "ed237dce23ef407fa74b615f8f6c914f"
format_stamp: "Formatted at 2026-06-19 18:47:40 on dist-test-slave-m9wg"
server_key: "8c344ee2d928dc2fc5e5819090cf426f"
server_key_iv: "4d14c19b4b0fad8aa5a8b2e3623d31f1"
server_key_version: "encryptionkey@0"
I20260619 18:47:40.890269 10763 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260619 18:47:40.893548 10861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:40.894855 10763 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 18:47:40.895000 10763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/master-2/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/master-2/wal
uuid: "ed237dce23ef407fa74b615f8f6c914f"
format_stamp: "Formatted at 2026-06-19 18:47:40 on dist-test-slave-m9wg"
server_key: "8c344ee2d928dc2fc5e5819090cf426f"
server_key_iv: "4d14c19b4b0fad8aa5a8b2e3623d31f1"
server_key_version: "encryptionkey@0"
I20260619 18:47:40.895164 10763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:47:40.913761 10763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:47:40.917894 10763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:47:40.918129 10763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:47:40.936324 10763 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.124:32923
I20260619 18:47:40.936338 10914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.124:32923 every 8 connection(s)
I20260619 18:47:40.937011 10900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:40.929714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45672 (local address 127.3.243.125:37445)
0619 18:47:40.930708 (+ 994us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:40.930712 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:40.930729 (+ 17us) server_negotiation.cc:407] Connection header received
0619 18:47:40.930790 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:40.930794 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:40.930860 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:40.930979 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:47:40.931554 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:40.933195 (+ 1641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:40.935126 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:40.936058 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:40.936238 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:40.936458 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:40.936667 (+ 209us) server_negotiation.cc:299] Negotiation successful
0619 18:47:40.936776 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":875,"thread_start_us":758,"threads_started":1}
I20260619 18:47:40.937072 10898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:40.929555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.125:37445 (local address 127.0.0.1:45672)
0619 18:47:40.930373 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:40.930396 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:40.930562 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:40.931057 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:40.931064 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:40.931098 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:47:40.931346 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:40.931361 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:40.933345 (+ 1984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:40.933352 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:40.934961 (+ 1609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:40.934975 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:40.936260 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:40.936267 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:40.936465 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:40.936531 (+ 66us) client_negotiation.cc:769] Sending connection context
0619 18:47:40.936643 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 18:47:40.936795 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":696,"thread_start_us":599,"threads_started":1}
I20260619 18:47:40.938083 10763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/master-2/data/info.pb
I20260619 18:47:40.938172 10807 raft_consensus.cc:1203] T 00000000000000000000000000000000 P ae1e1640d5134c40a77de2ae1fd29530 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260619 18:47:40.938794 10835 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae1e1640d5134c40a77de2ae1fd29530 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e612a9d44274aadb7de143295bf9e38. Latest consensus state: current_term: 1 leader_uuid: "3e612a9d44274aadb7de143295bf9e38" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3e612a9d44274aadb7de143295bf9e38" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 41477 } } peers { permanent_uuid: "ae1e1640d5134c40a77de2ae1fd29530" member_type: NON_VOTER last_known_addr { host: "127.3.243.125" port: 37445 } attrs { promote: true } } }
I20260619 18:47:40.939091 10835 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae1e1640d5134c40a77de2ae1fd29530 [sys.catalog]: This master's current role is: LEARNER
I20260619 18:47:40.939761 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 10763
I20260619 18:47:40.942242 10915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:47:40.950847 10915 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.243.126" port: 41477 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 18:47:40 dist-test-slave-m9wg krb5kdc[10155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894857, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.243.124@KRBTEST.COM
Jun 19 18:47:40 dist-test-slave-m9wg krb5kdc[10155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894860, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.243.124@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM
I20260619 18:47:40.962599 10918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:40.941733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39866 (local address 127.3.243.124:32923)
0619 18:47:40.942455 (+ 722us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:40.942468 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:40.942518 (+ 50us) server_negotiation.cc:407] Connection header received
0619 18:47:40.944664 (+ 2146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:40.944689 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:40.945174 (+ 485us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:40.945616 (+ 442us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:40.946817 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:40.948762 (+ 1945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:40.950464 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:40.951130 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:40.951268 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:40.955864 (+ 4596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:40.955871 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:40.955880 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:40.955915 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:40.958610 (+ 2695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:40.959362 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:40.959367 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:40.959373 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:40.959527 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:40.959892 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:40.959895 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:40.959896 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:40.960396 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:40.960659 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:40.961091 (+ 432us) server_negotiation.cc:299] Negotiation successful
0619 18:47:40.961452 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":504,"thread_start_us":296,"threads_started":1}
I20260619 18:47:40.966792 10919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:40.953501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:41477 (local address 127.0.0.1:32998)
0619 18:47:40.954233 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:40.954270 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:40.954427 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:40.954760 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:40.954765 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:40.955333 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:47:40.955557 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:40.955565 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:40.956601 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:40.956606 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:40.957729 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:40.957740 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:40.958411 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:40.958418 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:40.958556 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:40.959075 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:47:40.959097 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:47:40.961167 (+ 2070us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:47:40.964425 (+ 3258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:40.964433 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:40.964438 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:40.964833 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:40.965191 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:40.965196 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:40.965200 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:40.965326 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:40.965807 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:47:40.965814 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:47:40.965964 (+ 150us) client_negotiation.cc:769] Sending connection context
0619 18:47:40.966118 (+ 154us) client_negotiation.cc:240] Negotiation successful
0619 18:47:40.966401 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":366,"threads_started":1}
I20260619 18:47:40.966802 10391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:40.953635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32998 (local address 127.3.243.126:41477)
0619 18:47:40.953817 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:40.953823 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:40.954379 (+ 556us) server_negotiation.cc:407] Connection header received
0619 18:47:40.954539 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:40.954545 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:40.954618 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:40.954720 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:40.955695 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:40.956473 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:40.957855 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:40.958230 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:40.958329 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:40.961425 (+ 3096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:40.961431 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:40.961436 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:40.961476 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:40.964318 (+ 2842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:40.964944 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:40.964948 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:40.964950 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:40.965096 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:40.965424 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:40.965429 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:40.965432 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:40.965690 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:40.965817 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:40.966248 (+ 431us) server_negotiation.cc:299] Negotiation successful
0619 18:47:40.966464 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 18:47:40.968123 10915 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.243.125" port: 37445 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 18:47:40 dist-test-slave-m9wg krb5kdc[10155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894860, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.243.124@KRBTEST.COM for oryx/127.3.243.125@KRBTEST.COM
I20260619 18:47:40.979139 10919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:40.968824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.125:37445 (local address 127.0.0.1:45676)
0619 18:47:40.968988 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:40.969004 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:40.969073 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:40.969320 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:40.969323 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:40.969619 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:47:40.969754 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:40.969762 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:40.970629 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:40.970633 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:40.971518 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:40.971529 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:40.972169 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:40.972175 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:40.972298 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:40.972741 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:47:40.972756 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:47:40.974469 (+ 1713us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:47:40.977245 (+ 2776us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:40.977249 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:40.977252 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:40.977616 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:40.977941 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:40.977943 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:40.977945 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:40.978061 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:40.978507 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:47:40.978510 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:47:40.978622 (+ 112us) client_negotiation.cc:769] Sending connection context
0619 18:47:40.978730 (+ 108us) client_negotiation.cc:240] Negotiation successful
0619 18:47:40.978923 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260619 18:47:40.979198 10900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:40.968918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45676 (local address 127.3.243.125:37445)
0619 18:47:40.969062 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:40.969066 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:40.969081 (+ 15us) server_negotiation.cc:407] Connection header received
0619 18:47:40.969156 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:40.969160 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:40.969226 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:40.969300 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:40.969852 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:40.970531 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:40.971648 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:40.972043 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:40.972132 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:40.974650 (+ 2518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:40.974655 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:40.974659 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:40.974691 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:40.977143 (+ 2452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:40.977714 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:40.977718 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:40.977720 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:40.977866 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:40.978132 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:40.978134 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:40.978136 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:40.978409 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:40.978536 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:40.978812 (+ 276us) server_negotiation.cc:299] Negotiation successful
0619 18:47:40.978982 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 18:47:40.980137 10915 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.243.124" port: 32923 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 18:47:40 dist-test-slave-m9wg krb5kdc[10155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894860, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.243.124@KRBTEST.COM for oryx/127.3.243.124@KRBTEST.COM
I20260619 18:47:40.990871 10919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:40.980774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.124:32923 (local address 127.0.0.1:39876)
0619 18:47:40.980964 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:40.980979 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:40.981060 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:40.981278 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:40.981281 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:40.981574 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:47:40.981711 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:40.981718 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:40.982563 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:40.982566 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:40.983383 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:40.983392 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:40.984001 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:40.984005 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:40.984143 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:40.984619 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:47:40.984634 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:47:40.986416 (+ 1782us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:47:40.989002 (+ 2586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:40.989005 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:40.989008 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:40.989322 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:40.989599 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:40.989602 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:40.989604 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:40.989748 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:40.990151 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:47:40.990154 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:47:40.990268 (+ 114us) client_negotiation.cc:769] Sending connection context
0619 18:47:40.990363 (+ 95us) client_negotiation.cc:240] Negotiation successful
0619 18:47:40.990537 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260619 18:47:40.990922 10918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:40.980827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39876 (local address 127.3.243.124:32923)
0619 18:47:40.980969 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:40.980973 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:40.981027 (+ 54us) server_negotiation.cc:407] Connection header received
0619 18:47:40.981132 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:40.981135 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:40.981185 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:40.981256 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:40.981813 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:40.982470 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:40.983486 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:40.983879 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:40.983964 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:40.986566 (+ 2602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:40.986570 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:40.986574 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:40.986600 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:40.988905 (+ 2305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:40.989408 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:40.989411 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:40.989413 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:40.989520 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:40.989825 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:40.989828 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:40.989830 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:40.990069 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:40.990168 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:40.990437 (+ 269us) server_negotiation.cc:299] Negotiation successful
0619 18:47:40.990649 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":51}
I20260619 18:47:40.995396 10915 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed237dce23ef407fa74b615f8f6c914f: Bootstrap starting.
I20260619 18:47:40.998502 10915 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed237dce23ef407fa74b615f8f6c914f: Neither blocks nor log segments found. Creating new log.
I20260619 18:47:40.999512 10915 log.cc:824] T 00000000000000000000000000000000 P ed237dce23ef407fa74b615f8f6c914f: Log is configured to *not* fsync() on all Append() calls
I20260619 18:47:41.002579 10915 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed237dce23ef407fa74b615f8f6c914f: No bootstrap required, opened a new log
I20260619 18:47:41.009451 10915 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed237dce23ef407fa74b615f8f6c914f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e612a9d44274aadb7de143295bf9e38" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 41477 } } peers { permanent_uuid: "ae1e1640d5134c40a77de2ae1fd29530" member_type: VOTER last_known_addr { host: "127.3.243.125" port: 37445 } } peers { permanent_uuid: "ed237dce23ef407fa74b615f8f6c914f" member_type: VOTER last_known_addr { host: "127.3.243.124" port: 32923 } }
I20260619 18:47:41.009738 10915 raft_consensus.cc:374] T 00000000000000000000000000000000 P ed237dce23ef407fa74b615f8f6c914f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:47:41.009821 10915 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed237dce23ef407fa74b615f8f6c914f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed237dce23ef407fa74b615f8f6c914f, State: Initialized, Role: FOLLOWER
I20260619 18:47:41.010327 10915 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed237dce23ef407fa74b615f8f6c914f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e612a9d44274aadb7de143295bf9e38" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 41477 } } peers { permanent_uuid: "ae1e1640d5134c40a77de2ae1fd29530" member_type: VOTER last_known_addr { host: "127.3.243.125" port: 37445 } } peers { permanent_uuid: "ed237dce23ef407fa74b615f8f6c914f" member_type: VOTER last_known_addr { host: "127.3.243.124" port: 32923 } }
I20260619 18:47:41.011071 10921 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed237dce23ef407fa74b615f8f6c914f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e612a9d44274aadb7de143295bf9e38" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 41477 } } peers { permanent_uuid: "ae1e1640d5134c40a77de2ae1fd29530" member_type: VOTER last_known_addr { host: "127.3.243.125" port: 37445 } } peers { permanent_uuid: "ed237dce23ef407fa74b615f8f6c914f" member_type: VOTER last_known_addr { host: "127.3.243.124" port: 32923 } } }
I20260619 18:47:41.011375 10921 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed237dce23ef407fa74b615f8f6c914f [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:47:41.011966 10915 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed237dce23ef407fa74b615f8f6c914f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:47:41.017091 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 10261
W20260619 18:47:41.021016 10932 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ed237dce23ef407fa74b615f8f6c914f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 18:47:41.021224 10932 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 18:47:41.024281 10763 master_runner.cc:428] Detected that this master ed237dce23ef407fa74b615f8f6c914f is joining an existing cluster
I20260619 18:47:41.024395 10763 master_runner.cc:432] Initiating AddMaster RPC to add 127.3.243.124:32923
Jun 19 18:47:41 dist-test-slave-m9wg krb5kdc[10155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894857, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.243.65@KRBTEST.COM
I20260619 18:47:41.041672 10934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:41.018135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50172 (local address 127.3.243.65:34185)
0619 18:47:41.021958 (+ 3823us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:41.021968 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:41.021998 (+ 30us) server_negotiation.cc:407] Connection header received
0619 18:47:41.022075 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:41.022084 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:41.022318 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:41.022582 (+ 264us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:41.023953 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:41.025105 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:41.026758 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:41.027292 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:41.027399 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:41.033404 (+ 6005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:41.033418 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:41.033433 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:41.033478 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:41.037082 (+ 3604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:41.037879 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:41.037888 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:41.037891 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:41.038039 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:41.038472 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:41.038477 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:41.038481 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:41.038885 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:41.040096 (+ 1211us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:41.040515 (+ 419us) server_negotiation.cc:299] Negotiation successful
0619 18:47:41.040979 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3659,"thread_start_us":3499,"threads_started":1}
I20260619 18:47:41.043179 10384 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:47:41.048231 10391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:41.028941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33008 (local address 127.3.243.126:41477)
0619 18:47:41.029099 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:41.029104 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:41.033896 (+ 4792us) server_negotiation.cc:407] Connection header received
0619 18:47:41.034151 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:41.034157 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:41.034233 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:41.034301 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:41.035761 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:41.036887 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:41.038855 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:41.039411 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:41.039491 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:41.041835 (+ 2344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:41.041841 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:41.041845 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:41.041881 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:41.044763 (+ 2882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:41.045567 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:41.045572 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:41.045575 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:41.045746 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:41.046321 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:41.046326 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:41.046329 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:41.046671 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:41.046875 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:41.047618 (+ 743us) server_negotiation.cc:299] Negotiation successful
0619 18:47:41.047852 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 18:47:41.049542 10940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:41.028575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:41477 (local address 127.0.0.1:33008)
0619 18:47:41.033767 (+ 5192us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:41.033785 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:41.034005 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:41.034506 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:41.034510 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:41.035133 (+ 623us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:47:41.035561 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:41.035574 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:41.037061 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:41.037065 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:41.038666 (+ 1601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:41.038682 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:41.039765 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:41.039771 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:41.039945 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:41.040509 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:47:41.040537 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:47:41.041611 (+ 1074us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:47:41.044937 (+ 3326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:41.044942 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:41.044945 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:41.045355 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:41.045943 (+ 588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:41.045947 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:41.045951 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:41.046124 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:41.047049 (+ 925us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:47:41.047054 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:47:41.047263 (+ 209us) client_negotiation.cc:769] Sending connection context
0619 18:47:41.048820 (+ 1557us) client_negotiation.cc:240] Negotiation successful
0619 18:47:41.049097 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5022,"thread_start_us":12103,"threads_started":1}
I20260619 18:47:41.051095 10900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:41.028790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45682 (local address 127.3.243.125:37445)
0619 18:47:41.028956 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:41.028961 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:41.034258 (+ 5297us) server_negotiation.cc:407] Connection header received
0619 18:47:41.034481 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:41.034487 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:41.034560 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:41.034629 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:41.036048 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:41.037132 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:41.039216 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:41.039871 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:41.039963 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:41.044874 (+ 4911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:41.044881 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:41.044884 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:41.044922 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:41.048225 (+ 3303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:41.048909 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:41.048914 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:41.048918 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:41.049063 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:41.049522 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:41.049528 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:41.049531 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:41.049864 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:41.049976 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:41.050543 (+ 567us) server_negotiation.cc:299] Negotiation successful
0619 18:47:41.050771 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 18:47:41.051996 10939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:41.028931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.125:37445 (local address 127.0.0.1:45682)
0619 18:47:41.034134 (+ 5203us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:41.034151 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:41.034348 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:41.034848 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:41.034853 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:41.035552 (+ 699us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:47:41.035866 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:41.035878 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:41.037343 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:41.037349 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:41.039056 (+ 1707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:41.039070 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:41.040256 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:41.040265 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:41.040426 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:41.042410 (+ 1984us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:47:41.042433 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:47:41.043431 (+ 998us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:47:41.048375 (+ 4944us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:41.048380 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:41.048383 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:41.048783 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:41.049210 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:41.049215 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:41.049223 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:41.049394 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:41.050063 (+ 669us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:47:41.050067 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:47:41.050286 (+ 219us) client_negotiation.cc:769] Sending connection context
0619 18:47:41.051266 (+ 980us) client_negotiation.cc:240] Negotiation successful
0619 18:47:41.051586 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5085,"thread_start_us":4506,"threads_started":1}
I20260619 18:47:41.063938 10918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:41.041244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39880 (local address 127.3.243.124:32923)
0619 18:47:41.041426 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:41.041432 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:41.050060 (+ 8628us) server_negotiation.cc:407] Connection header received
0619 18:47:41.050353 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:41.050358 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:41.050419 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:41.050474 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:41.052056 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:41.052896 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:41.055256 (+ 2360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:41.055703 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:41.055801 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:41.057739 (+ 1938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:41.057745 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:41.057750 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:41.057781 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:41.061257 (+ 3476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:41.061993 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:41.061997 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:41.062000 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:41.062155 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:41.062518 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:41.062523 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:41.062526 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:41.062832 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:41.062968 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:41.063407 (+ 439us) server_negotiation.cc:299] Negotiation successful
0619 18:47:41.063603 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
Jun 19 18:47:41 dist-test-slave-m9wg krb5kdc[10155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894858, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.243.126@KRBTEST.COM for oryx/127.3.243.124@KRBTEST.COM
I20260619 18:47:41.075088 10940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:41.041537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.124:32923 (local address 127.0.0.1:39880)
0619 18:47:41.049996 (+ 8459us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:41.050013 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:41.050179 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:41.050794 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:41.050798 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:41.051600 (+ 802us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:47:41.051852 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:41.051863 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:41.054216 (+ 2353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:41.054220 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:41.055130 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:41.055141 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:41.055823 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:41.055829 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:41.055975 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:41.056529 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:47:41.056551 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:47:41.057554 (+ 1003us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:47:41.061398 (+ 3844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:41.061403 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:41.061407 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:41.061883 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:41.062272 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:41.062277 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:41.062280 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:41.062419 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:41.063041 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:47:41.063046 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:47:41.063167 (+ 121us) client_negotiation.cc:769] Sending connection context
0619 18:47:41.074373 (+ 11206us) client_negotiation.cc:240] Negotiation successful
0619 18:47:41.074665 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8373,"thread_start_us":14806,"threads_started":1}
I20260619 18:47:41.081094 10898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:41.061506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.124:32923 (local address 127.0.0.1:39886)
0619 18:47:41.061853 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:41.061869 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:41.061966 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:41.064816 (+ 2850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:41.064821 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:41.065238 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:47:41.065411 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:41.065422 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:41.067130 (+ 1708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:41.067136 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:41.068542 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:41.068557 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:41.069596 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:41.069602 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:41.069782 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:41.070348 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:47:41.070369 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:47:41.072989 (+ 2620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:47:41.078547 (+ 5558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:41.078553 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:41.078557 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:41.079045 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:41.079418 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:41.079423 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:41.079427 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:41.079574 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:41.080140 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:47:41.080145 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:47:41.080327 (+ 182us) client_negotiation.cc:769] Sending connection context
0619 18:47:41.080486 (+ 159us) client_negotiation.cc:240] Negotiation successful
0619 18:47:41.080748 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237}
I20260619 18:47:41.081386 10918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:41.062160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39886 (local address 127.3.243.124:32923)
0619 18:47:41.064459 (+ 2299us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:41.064465 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:41.064481 (+ 16us) server_negotiation.cc:407] Connection header received
0619 18:47:41.064566 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:41.064571 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:41.064644 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:41.065551 (+ 907us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:41.065742 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:41.066976 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:41.068727 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:41.069428 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:41.073143 (+ 3715us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:41.073318 (+ 175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:41.073323 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:41.073328 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:41.073359 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:41.078396 (+ 5037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:41.079162 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:41.079167 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:41.079171 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:41.079313 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:41.079680 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:41.079685 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:41.079688 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:41.080027 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:41.080195 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:41.080596 (+ 401us) server_negotiation.cc:299] Negotiation successful
0619 18:47:41.080819 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2199,"thread_start_us":12199,"threads_started":1}
I20260619 18:47:41.082741 10200 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.3.243.124:32923: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "ed237dce23ef407fa74b615f8f6c914f"
member_type: NON_VOTER
last_known_addr {
host: "127.3.243.124"
port: 32923
}
attrs {
promote: true
}
}
dest_uuid: "3e612a9d44274aadb7de143295bf9e38"
cas_config_opid_index: 5
I20260619 18:47:41.083814 10200 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e612a9d44274aadb7de143295bf9e38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3e612a9d44274aadb7de143295bf9e38" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 41477 } } peers { permanent_uuid: "ae1e1640d5134c40a77de2ae1fd29530" member_type: NON_VOTER last_known_addr { host: "127.3.243.125" port: 37445 } attrs { promote: true } } peers { permanent_uuid: "ed237dce23ef407fa74b615f8f6c914f" member_type: NON_VOTER last_known_addr { host: "127.3.243.124" port: 32923 } attrs { promote: true } }
I20260619 18:47:41.085829 10756 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3e612a9d44274aadb7de143295bf9e38 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER ed237dce23ef407fa74b615f8f6c914f (127.3.243.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3e612a9d44274aadb7de143295bf9e38" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 41477 } } peers { permanent_uuid: "ae1e1640d5134c40a77de2ae1fd29530" member_type: NON_VOTER last_known_addr { host: "127.3.243.125" port: 37445 } attrs { promote: true } } peers { permanent_uuid: "ed237dce23ef407fa74b615f8f6c914f" member_type: NON_VOTER last_known_addr { host: "127.3.243.124" port: 32923 } attrs { promote: true } } }
I20260619 18:47:41.089147 10807 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ae1e1640d5134c40a77de2ae1fd29530 [term 1 LEARNER]: Refusing update from remote peer 3e612a9d44274aadb7de143295bf9e38: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260619 18:47:41.090168 10756 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.3.243.124:32923
I20260619 18:47:41.090485 10888 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed237dce23ef407fa74b615f8f6c914f [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:47:41.094137 10888 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ed237dce23ef407fa74b615f8f6c914f [term 1 FOLLOWER]: Refusing update from remote peer 3e612a9d44274aadb7de143295bf9e38: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260619 18:47:41.094204 10763 master.cc:537] Master@127.3.243.124:32923 shutting down...
I20260619 18:47:41.105055 10763 raft_consensus.cc:2229] T 00000000000000000000000000000000 P ed237dce23ef407fa74b615f8f6c914f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:47:41.105469 10763 raft_consensus.cc:2258] T 00000000000000000000000000000000 P ed237dce23ef407fa74b615f8f6c914f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:47:41.105722 10763 tablet_replica.cc:330] T 00000000000000000000000000000000 P ed237dce23ef407fa74b615f8f6c914f: stopping tablet replica
I20260619 18:47:41.111920 10945 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3e612a9d44274aadb7de143295bf9e38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae1e1640d5134c40a77de2ae1fd29530" member_type: NON_VOTER last_known_addr { host: "127.3.243.125" port: 37445 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260619 18:47:41.114930 10807 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ae1e1640d5134c40a77de2ae1fd29530 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER ed237dce23ef407fa74b615f8f6c914f (127.3.243.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3e612a9d44274aadb7de143295bf9e38" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 41477 } } peers { permanent_uuid: "ae1e1640d5134c40a77de2ae1fd29530" member_type: NON_VOTER last_known_addr { host: "127.3.243.125" port: 37445 } attrs { promote: true } } peers { permanent_uuid: "ed237dce23ef407fa74b615f8f6c914f" member_type: NON_VOTER last_known_addr { host: "127.3.243.124" port: 32923 } attrs { promote: true } } }
I20260619 18:47:41.119340 10835 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae1e1640d5134c40a77de2ae1fd29530 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3e612a9d44274aadb7de143295bf9e38" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3e612a9d44274aadb7de143295bf9e38" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 41477 } } peers { permanent_uuid: "ae1e1640d5134c40a77de2ae1fd29530" member_type: NON_VOTER last_known_addr { host: "127.3.243.125" port: 37445 } attrs { promote: true } } peers { permanent_uuid: "ed237dce23ef407fa74b615f8f6c914f" member_type: NON_VOTER last_known_addr { host: "127.3.243.124" port: 32923 } attrs { promote: true } } }
I20260619 18:47:41.119606 10835 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae1e1640d5134c40a77de2ae1fd29530 [sys.catalog]: This master's current role is: LEARNER
I20260619 18:47:41.120967 10763 master.cc:552] Master@127.3.243.124:32923 shutdown complete.
I20260619 18:47:41.121047 10763 master_runner.cc:305] Clearing existing system tablet
I20260619 18:47:41.121421 10763 fs_manager.cc:359] Metadata directory not provided
I20260619 18:47:41.121524 10763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:47:41.128830 10946 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3e612a9d44274aadb7de143295bf9e38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed237dce23ef407fa74b615f8f6c914f" member_type: NON_VOTER last_known_addr { host: "127.3.243.124" port: 32923 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:47:41.129604 10763 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.002s sys 0.001s
I20260619 18:47:41.131373 10898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:41.130889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.124:32923 (local address 127.0.0.1:39898)
0619 18:47:41.131063 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:41.131200 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.124:32923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260619 18:47:41.134420 10756 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 3e612a9d44274aadb7de143295bf9e38: attempting to promote NON_VOTER ae1e1640d5134c40a77de2ae1fd29530 to VOTER
I20260619 18:47:41.135201 10756 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e612a9d44274aadb7de143295bf9e38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3e612a9d44274aadb7de143295bf9e38" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 41477 } } peers { permanent_uuid: "ae1e1640d5134c40a77de2ae1fd29530" member_type: VOTER last_known_addr { host: "127.3.243.125" port: 37445 } attrs { promote: false } } peers { permanent_uuid: "ed237dce23ef407fa74b615f8f6c914f" member_type: NON_VOTER last_known_addr { host: "127.3.243.124" port: 32923 } attrs { promote: true } }
I20260619 18:47:41.136039 10949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:41.137550 10945 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e612a9d44274aadb7de143295bf9e38 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "3e612a9d44274aadb7de143295bf9e38" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3e612a9d44274aadb7de143295bf9e38" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 41477 } } peers { permanent_uuid: "ae1e1640d5134c40a77de2ae1fd29530" member_type: NON_VOTER last_known_addr { host: "127.3.243.125" port: 37445 } attrs { promote: true } } peers { permanent_uuid: "ed237dce23ef407fa74b615f8f6c914f" member_type: NON_VOTER last_known_addr { host: "127.3.243.124" port: 32923 } attrs { promote: true } } } pending_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3e612a9d44274aadb7de143295bf9e38" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 41477 } } peers { permanent_uuid: "ae1e1640d5134c40a77de2ae1fd29530" member_type: VOTER last_known_addr { host: "127.3.243.125" port: 37445 } attrs { promote: false } } peers { permanent_uuid: "ed237dce23ef407fa74b615f8f6c914f" member_type: NON_VOTER last_known_addr { host: "127.3.243.124" port: 32923 } attrs { promote: true } } }
I20260619 18:47:41.137970 10945 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e612a9d44274aadb7de143295bf9e38 [sys.catalog]: This master's current role is: LEADER
I20260619 18:47:41.138974 10898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:41.138591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.124:32923 (local address 127.0.0.1:39900)
0619 18:47:41.138753 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:41.138826 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.124:32923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260619 18:47:41.140784 10763 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:47:41.141789 10898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:41.141454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.124:32923 (local address 127.0.0.1:39904)
0619 18:47:41.141568 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:41.141632 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.124:32923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260619 18:47:41.142838 10898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:41.142527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.124:32923 (local address 127.0.0.1:39910)
0619 18:47:41.142664 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:41.142725 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.124:32923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260619 18:47:41.143110 10186 consensus_peers.cc:487] T 00000000000000000000000000000000 P 3e612a9d44274aadb7de143295bf9e38 -> Peer ed237dce23ef407fa74b615f8f6c914f (127.3.243.124:32923): Couldn't send request to peer ed237dce23ef407fa74b615f8f6c914f. Status: Network error: Client connection negotiation failed: client connection to 127.3.243.124:32923: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260619 18:47:41.143435 10763 fs_manager.cc:725] Time spent opening block manager: real 0.013s user 0.002s sys 0.001s
I20260619 18:47:41.144181 10951 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e612a9d44274aadb7de143295bf9e38 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3e612a9d44274aadb7de143295bf9e38" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3e612a9d44274aadb7de143295bf9e38" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 41477 } } peers { permanent_uuid: "ae1e1640d5134c40a77de2ae1fd29530" member_type: NON_VOTER last_known_addr { host: "127.3.243.125" port: 37445 } attrs { promote: true } } peers { permanent_uuid: "ed237dce23ef407fa74b615f8f6c914f" member_type: NON_VOTER last_known_addr { host: "127.3.243.124" port: 32923 } attrs { promote: true } } } pending_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3e612a9d44274aadb7de143295bf9e38" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 41477 } } peers { permanent_uuid: "ae1e1640d5134c40a77de2ae1fd29530" member_type: VOTER last_known_addr { host: "127.3.243.125" port: 37445 } attrs { promote: false } } peers { permanent_uuid: "ed237dce23ef407fa74b615f8f6c914f" member_type: NON_VOTER last_known_addr { host: "127.3.243.124" port: 32923 } attrs { promote: true } } }
I20260619 18:47:41.144584 10807 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ae1e1640d5134c40a77de2ae1fd29530 [term 1 LEARNER]: Refusing update from remote peer 3e612a9d44274aadb7de143295bf9e38: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260619 18:47:41.144742 10763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/master-2/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/master-2/wal
uuid: "ed237dce23ef407fa74b615f8f6c914f"
format_stamp: "Formatted at 2026-06-19 18:47:40 on dist-test-slave-m9wg"
server_key: "8c344ee2d928dc2fc5e5819090cf426f"
server_key_iv: "4d14c19b4b0fad8aa5a8b2e3623d31f1"
server_key_version: "encryptionkey@0"
I20260619 18:47:41.145184 10945 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3e612a9d44274aadb7de143295bf9e38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae1e1640d5134c40a77de2ae1fd29530" member_type: VOTER last_known_addr { host: "127.3.243.125" port: 37445 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260619 18:47:41.149753 10763 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P ed237dce23ef407fa74b615f8f6c914f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:47:41.151203 10807 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ae1e1640d5134c40a77de2ae1fd29530 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, ae1e1640d5134c40a77de2ae1fd29530 (127.3.243.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3e612a9d44274aadb7de143295bf9e38" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 41477 } } peers { permanent_uuid: "ae1e1640d5134c40a77de2ae1fd29530" member_type: VOTER last_known_addr { host: "127.3.243.125" port: 37445 } attrs { promote: false } } peers { permanent_uuid: "ed237dce23ef407fa74b615f8f6c914f" member_type: NON_VOTER last_known_addr { host: "127.3.243.124" port: 32923 } attrs { promote: true } } }
I20260619 18:47:41.155153 10835 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae1e1640d5134c40a77de2ae1fd29530 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3e612a9d44274aadb7de143295bf9e38" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3e612a9d44274aadb7de143295bf9e38" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 41477 } } peers { permanent_uuid: "ae1e1640d5134c40a77de2ae1fd29530" member_type: VOTER last_known_addr { host: "127.3.243.125" port: 37445 } attrs { promote: false } } peers { permanent_uuid: "ed237dce23ef407fa74b615f8f6c914f" member_type: NON_VOTER last_known_addr { host: "127.3.243.124" port: 32923 } attrs { promote: true } } }
I20260619 18:47:41.155380 10835 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae1e1640d5134c40a77de2ae1fd29530 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:47:41.156100 10946 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3e612a9d44274aadb7de143295bf9e38 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, ae1e1640d5134c40a77de2ae1fd29530 (127.3.243.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3e612a9d44274aadb7de143295bf9e38" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 41477 } } peers { permanent_uuid: "ae1e1640d5134c40a77de2ae1fd29530" member_type: VOTER last_known_addr { host: "127.3.243.125" port: 37445 } attrs { promote: false } } peers { permanent_uuid: "ed237dce23ef407fa74b615f8f6c914f" member_type: NON_VOTER last_known_addr { host: "127.3.243.124" port: 32923 } attrs { promote: true } } }
I20260619 18:47:41.156647 10763 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P ed237dce23ef407fa74b615f8f6c914f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 18:47:41.156822 10763 log.cc:1196] T 00000000000000000000000000000000 P ed237dce23ef407fa74b615f8f6c914f: Deleting WAL directory at /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260619 18:47:41.157274 10763 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P ed237dce23ef407fa74b615f8f6c914f: Deleting consensus metadata
I20260619 18:47:41.157534 10763 master_runner.cc:315] Copying system tablet from 127.3.243.126:41477
I20260619 18:47:41.159756 10951 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e612a9d44274aadb7de143295bf9e38 [sys.catalog]: This master's current role is: LEADER
I20260619 18:47:41.160530 10951 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e612a9d44274aadb7de143295bf9e38 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "3e612a9d44274aadb7de143295bf9e38" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3e612a9d44274aadb7de143295bf9e38" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 41477 } } peers { permanent_uuid: "ae1e1640d5134c40a77de2ae1fd29530" member_type: VOTER last_known_addr { host: "127.3.243.125" port: 37445 } attrs { promote: false } } peers { permanent_uuid: "ed237dce23ef407fa74b615f8f6c914f" member_type: NON_VOTER last_known_addr { host: "127.3.243.124" port: 32923 } attrs { promote: true } } }
I20260619 18:47:41.160768 10951 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e612a9d44274aadb7de143295bf9e38 [sys.catalog]: This master's current role is: LEADER
I20260619 18:47:41.161044 10946 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e612a9d44274aadb7de143295bf9e38 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3e612a9d44274aadb7de143295bf9e38" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3e612a9d44274aadb7de143295bf9e38" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 41477 } } peers { permanent_uuid: "ae1e1640d5134c40a77de2ae1fd29530" member_type: VOTER last_known_addr { host: "127.3.243.125" port: 37445 } attrs { promote: false } } peers { permanent_uuid: "ed237dce23ef407fa74b615f8f6c914f" member_type: NON_VOTER last_known_addr { host: "127.3.243.124" port: 32923 } attrs { promote: true } } }
I20260619 18:47:41.161249 10946 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e612a9d44274aadb7de143295bf9e38 [sys.catalog]: This master's current role is: LEADER
I20260619 18:47:41.163132 10763 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P ed237dce23ef407fa74b615f8f6c914f: tablet copy: Beginning tablet copy session from remote peer at address 127.3.243.126:41477
I20260619 18:47:41.179718 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 10261
I20260619 18:47:41.181821 10391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:41.163850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33020 (local address 127.3.243.126:41477)
0619 18:47:41.164060 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:41.164066 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:41.164704 (+ 638us) server_negotiation.cc:407] Connection header received
0619 18:47:41.164895 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:41.164901 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:41.164979 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:41.165074 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:41.166094 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:41.167236 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:41.169181 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:41.169795 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:41.169894 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:41.172369 (+ 2475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:41.172376 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:41.172380 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:41.172421 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:41.178303 (+ 5882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:41.179096 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:41.179102 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:41.179105 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:41.179264 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:41.179733 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:41.179738 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:41.179741 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:41.180074 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:41.180235 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:41.181203 (+ 968us) server_negotiation.cc:299] Negotiation successful
0619 18:47:41.181423 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260619 18:47:41.183041 10959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:41.163864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:41477 (local address 127.0.0.1:33020)
0619 18:47:41.164576 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:41.164601 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:41.164755 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:41.165126 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:41.165131 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:41.165672 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:47:41.165946 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:41.165959 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:41.167416 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:41.167422 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:41.169004 (+ 1582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:41.169020 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:41.170177 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:41.170183 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:41.170381 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:41.171037 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:47:41.171061 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:47:41.172121 (+ 1060us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:47:41.178466 (+ 6345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:41.178472 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:41.178476 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:41.178936 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:41.179414 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:41.179419 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:41.179423 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:41.179589 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:41.180413 (+ 824us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:47:41.180418 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:47:41.180902 (+ 484us) client_negotiation.cc:769] Sending connection context
0619 18:47:41.182277 (+ 1375us) client_negotiation.cc:240] Negotiation successful
0619 18:47:41.182613 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":427,"threads_started":1}
I20260619 18:47:41.183898 10220 tablet_copy_service.cc:140] P 3e612a9d44274aadb7de143295bf9e38: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer ed237dce23ef407fa74b615f8f6c914f ({username='oryx', principal='oryx/127.3.243.124@KRBTEST.COM'} at 127.0.0.1:33020)
I20260619 18:47:41.184067 10220 tablet_copy_service.cc:161] P 3e612a9d44274aadb7de143295bf9e38: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer ed237dce23ef407fa74b615f8f6c914f at {username='oryx', principal='oryx/127.3.243.124@KRBTEST.COM'} at 127.0.0.1:33020: session id = ed237dce23ef407fa74b615f8f6c914f-00000000000000000000000000000000
I20260619 18:47:41.187665 10220 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 3e612a9d44274aadb7de143295bf9e38: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 18:47:41.189611 10763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 18:47:41.196616 10763 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P ed237dce23ef407fa74b615f8f6c914f: tablet copy: Starting download of 0 data blocks...
I20260619 18:47:41.196928 10763 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P ed237dce23ef407fa74b615f8f6c914f: tablet copy: Starting download of 1 WAL segments...
I20260619 18:47:41.199506 10763 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P ed237dce23ef407fa74b615f8f6c914f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 18:47:41.200285 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.65:34185
--local_ip_for_outbound_sockets=127.3.243.65
--tserver_master_addrs=127.3.243.126:41477
--webserver_port=44313
--webserver_interface=127.3.243.65
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/krb5kdc/oryx_127.3.243.65.keytab
--principal=oryx/127.3.243.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:37789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.3.243.126:41477,127.3.243.125:37445
--tserver_master_addrs=127.3.243.126:41477,127.3.243.125:37445,127.3.243.124:32923 with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 18:47:41.203898 10220 tablet_copy_service.cc:342] P 3e612a9d44274aadb7de143295bf9e38: Request end of tablet copy session ed237dce23ef407fa74b615f8f6c914f-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.3.243.124@KRBTEST.COM'} at 127.0.0.1:33020
I20260619 18:47:41.204032 10220 tablet_copy_service.cc:433] P 3e612a9d44274aadb7de143295bf9e38: ending tablet copy session ed237dce23ef407fa74b615f8f6c914f-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer ed237dce23ef407fa74b615f8f6c914f
I20260619 18:47:41.215696 10763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:47:41.218379 10961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:41.236873 10962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:41.239991 10964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:47:41.240554 10763 server_base.cc:1034] running on GCE node
I20260619 18:47:41.240886 10763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:47:41.241508 10763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:47:41.242693 10763 hybrid_clock.cc:648] HybridClock initialized: now 1781894861242641 us; error 32 us; skew 500 ppm
I20260619 18:47:41.242923 10763 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:47:41 dist-test-slave-m9wg krb5kdc[10155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894861, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.243.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:47:41.245009 10763 init.cc:377] Logged in from keytab as oryx/127.3.243.124@KRBTEST.COM (short username oryx)
I20260619 18:47:41.247941 10763 webserver.cc:460] Webserver started at http://127.3.243.124:43217/ using document root <none> and password file <none>
I20260619 18:47:41.248283 10763 fs_manager.cc:359] Metadata directory not provided
I20260619 18:47:41.248383 10763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:47:41.251374 10763 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 18:47:41.253392 10971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:41.253837 10763 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 18:47:41.253994 10763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/master-2/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/master-2/wal
uuid: "ed237dce23ef407fa74b615f8f6c914f"
format_stamp: "Formatted at 2026-06-19 18:47:40 on dist-test-slave-m9wg"
server_key: "8c344ee2d928dc2fc5e5819090cf426f"
server_key_iv: "4d14c19b4b0fad8aa5a8b2e3623d31f1"
server_key_version: "encryptionkey@0"
I20260619 18:47:41.254236 10763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:47:41.275552 10763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:47:41.279589 10763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:47:41.309899 10763 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.124:32923
I20260619 18:47:41.309984 11022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.124:32923 every 8 connection(s)
I20260619 18:47:41.310986 10763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/master-2/data/info.pb
I20260619 18:47:41.315992 11023 sys_catalog.cc:263] Verifying existing consensus state
I20260619 18:47:41.320062 11023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed237dce23ef407fa74b615f8f6c914f: Bootstrap starting.
I20260619 18:47:41.338531 11023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed237dce23ef407fa74b615f8f6c914f: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 18:47:41.338987 11023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed237dce23ef407fa74b615f8f6c914f: Bootstrap complete.
I20260619 18:47:41.340008 11023 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed237dce23ef407fa74b615f8f6c914f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3e612a9d44274aadb7de143295bf9e38" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 41477 } } peers { permanent_uuid: "ae1e1640d5134c40a77de2ae1fd29530" member_type: VOTER last_known_addr { host: "127.3.243.125" port: 37445 } attrs { promote: false } } peers { permanent_uuid: "ed237dce23ef407fa74b615f8f6c914f" member_type: NON_VOTER last_known_addr { host: "127.3.243.124" port: 32923 } attrs { promote: true } }
I20260619 18:47:41.340313 11023 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed237dce23ef407fa74b615f8f6c914f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ed237dce23ef407fa74b615f8f6c914f, State: Initialized, Role: LEARNER
I20260619 18:47:41.340590 11023 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed237dce23ef407fa74b615f8f6c914f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3e612a9d44274aadb7de143295bf9e38" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 41477 } } peers { permanent_uuid: "ae1e1640d5134c40a77de2ae1fd29530" member_type: VOTER last_known_addr { host: "127.3.243.125" port: 37445 } attrs { promote: false } } peers { permanent_uuid: "ed237dce23ef407fa74b615f8f6c914f" member_type: NON_VOTER last_known_addr { host: "127.3.243.124" port: 32923 } attrs { promote: true } }
I20260619 18:47:41.341364 11026 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed237dce23ef407fa74b615f8f6c914f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3e612a9d44274aadb7de143295bf9e38" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 41477 } } peers { permanent_uuid: "ae1e1640d5134c40a77de2ae1fd29530" member_type: VOTER last_known_addr { host: "127.3.243.125" port: 37445 } attrs { promote: false } } peers { permanent_uuid: "ed237dce23ef407fa74b615f8f6c914f" member_type: NON_VOTER last_known_addr { host: "127.3.243.124" port: 32923 } attrs { promote: true } } }
I20260619 18:47:41.341858 11026 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed237dce23ef407fa74b615f8f6c914f [sys.catalog]: This master's current role is: LEARNER
I20260619 18:47:41.341638 11023 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed237dce23ef407fa74b615f8f6c914f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:47:41.352548 11037 catalog_manager.cc:1259] Loaded cluster ID: 301fc5ce1f33458494634dbfacabe160
I20260619 18:47:41.352646 11037 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ed237dce23ef407fa74b615f8f6c914f: loading cluster ID for follower catalog manager: success
I20260619 18:47:41.356312 11037 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ed237dce23ef407fa74b615f8f6c914f: acquiring CA information for follower catalog manager: success
I20260619 18:47:41.358686 11037 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ed237dce23ef407fa74b615f8f6c914f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 18:47:41.564121 11038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:41.555925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39920 (local address 127.3.243.124:32923)
0619 18:47:41.556822 (+ 897us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:41.556826 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:41.556843 (+ 17us) server_negotiation.cc:407] Connection header received
0619 18:47:41.556906 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:41.556910 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:41.556978 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:41.557080 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:47:41.557568 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:41.559015 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:41.561625 (+ 2610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:41.562872 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:41.562970 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:41.563218 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:41.563604 (+ 386us) server_negotiation.cc:299] Negotiation successful
0619 18:47:41.563740 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":779,"thread_start_us":547,"threads_started":1}
I20260619 18:47:41.564198 10898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:41.555771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.124:32923 (local address 127.0.0.1:39920)
0619 18:47:41.555983 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:41.556018 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:41.556191 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:41.557137 (+ 946us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:41.557143 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:41.557176 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:47:41.557402 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:41.557416 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:41.559179 (+ 1763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:41.559184 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:41.561489 (+ 2305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:41.561506 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:41.563063 (+ 1557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:41.563069 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:41.563421 (+ 352us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:41.563461 (+ 40us) client_negotiation.cc:769] Sending connection context
0619 18:47:41.563565 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 18:47:41.563731 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260619 18:47:41.565383 10998 raft_consensus.cc:1203] T 00000000000000000000000000000000 P ed237dce23ef407fa74b615f8f6c914f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260619 18:47:41.566201 11026 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed237dce23ef407fa74b615f8f6c914f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e612a9d44274aadb7de143295bf9e38. Latest consensus state: current_term: 1 leader_uuid: "3e612a9d44274aadb7de143295bf9e38" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3e612a9d44274aadb7de143295bf9e38" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 41477 } } peers { permanent_uuid: "ae1e1640d5134c40a77de2ae1fd29530" member_type: VOTER last_known_addr { host: "127.3.243.125" port: 37445 } attrs { promote: false } } peers { permanent_uuid: "ed237dce23ef407fa74b615f8f6c914f" member_type: NON_VOTER last_known_addr { host: "127.3.243.124" port: 32923 } attrs { promote: true } } }
I20260619 18:47:41.566560 11026 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed237dce23ef407fa74b615f8f6c914f [sys.catalog]: This master's current role is: LEARNER
W20260619 18:47:41.603106 10960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:47:41.603473 10960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:47:41.603685 10960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:47:41.613488 10960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:47:41.613658 10960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:47:41.613873 10960 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.65
I20260619 18:47:41.626847 10960 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:37789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/krb5kdc/oryx_127.3.243.65.keytab
--principal=oryx/127.3.243.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.65:34185
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.243.65
--webserver_port=44313
--tserver_master_addrs=127.3.243.126:41477,127.3.243.125:37445,127.3.243.124:32923
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.65
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:47:41.628366 10960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:47:41.629828 10960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:47:41.645663 11043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:41.646248 11046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:41.646337 11044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:47:41.647675 10960 server_base.cc:1034] running on GCE node
I20260619 18:47:41.648409 10960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:47:41.649434 10960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:47:41.650662 10960 hybrid_clock.cc:648] HybridClock initialized: now 1781894861650579 us; error 43 us; skew 500 ppm
I20260619 18:47:41.651042 10960 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:47:41 dist-test-slave-m9wg krb5kdc[10155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894861, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.243.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:47:41.655164 10960 init.cc:377] Logged in from keytab as oryx/127.3.243.65@KRBTEST.COM (short username oryx)
I20260619 18:47:41.657313 10960 webserver.cc:460] Webserver started at http://127.3.243.65:44313/ using document root <none> and password file <none>
I20260619 18:47:41.658150 10960 fs_manager.cc:359] Metadata directory not provided
I20260619 18:47:41.658350 10960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:47:41.665161 10960 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 18:47:41.668669 11053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:41.670091 10960 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 18:47:41.670286 10960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/ts-0/wal
uuid: "69da4e79b53c43fc8becd20ad510c8b8"
format_stamp: "Formatted at 2026-06-19 18:47:38 on dist-test-slave-m9wg"
server_key: "1bd46fb8a572e51e756e0453120ab80a"
server_key_iv: "fa2989ee24f4672da138559764e9ac6e"
server_key_version: "encryptionkey@0"
I20260619 18:47:41.671518 10960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:47:41.712133 10960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:47:41.717104 10960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:47:41.717463 10960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:47:41.718922 10960 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:47:41.720625 10960 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:47:41.720789 10960 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:41.720925 10960 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:47:41.721057 10960 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:41.786765 10960 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.65:34185
I20260619 18:47:41.786929 11165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.65:34185 every 8 connection(s)
I20260619 18:47:41.788535 10960 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/ts-0/data/info.pb
I20260619 18:47:41.797420 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 10960
I20260619 18:47:41.797856 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 10395
Jun 19 18:47:41 dist-test-slave-m9wg krb5kdc[10155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894861, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.243.65@KRBTEST.COM for oryx/127.3.243.124@KRBTEST.COM
Jun 19 18:47:41 dist-test-slave-m9wg krb5kdc[10155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894861, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.243.65@KRBTEST.COM for oryx/127.3.243.125@KRBTEST.COM
Jun 19 18:47:41 dist-test-slave-m9wg krb5kdc[10155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894861, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.243.65@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM
I20260619 18:47:41.826900 11170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:41.792332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.65:57595 (local address 127.3.243.125:37445)
0619 18:47:41.793288 (+ 956us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:41.793293 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:41.795032 (+ 1739us) server_negotiation.cc:407] Connection header received
0619 18:47:41.800099 (+ 5067us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:41.800106 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:41.800272 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:41.800356 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:41.807430 (+ 7074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:41.808991 (+ 1561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:41.811494 (+ 2503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:41.812392 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:41.812820 (+ 428us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:41.817974 (+ 5154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:41.817981 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:41.817986 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:41.818024 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:41.822604 (+ 4580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:41.823634 (+ 1030us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:41.823639 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:41.823642 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:41.823838 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:41.824529 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:41.824534 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:41.824538 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:41.824930 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:41.825052 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:41.826148 (+ 1096us) server_negotiation.cc:299] Negotiation successful
0619 18:47:41.826377 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":830,"thread_start_us":678,"threads_started":1}
I20260619 18:47:41.826908 11038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:41.792954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.65:45369 (local address 127.3.243.124:32923)
0619 18:47:41.793880 (+ 926us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:41.793886 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:41.795066 (+ 1180us) server_negotiation.cc:407] Connection header received
0619 18:47:41.798992 (+ 3926us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:41.798998 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:41.799095 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:41.799208 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:41.802960 (+ 3752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:41.804375 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:41.810052 (+ 5677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:41.811042 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:41.811146 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:41.817966 (+ 6820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:41.817973 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:41.817978 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:41.818020 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:41.822444 (+ 4424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:41.823717 (+ 1273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:41.823723 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:41.823727 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:41.823925 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:41.824836 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:41.824841 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:41.824844 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:41.825250 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:41.825380 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:41.826144 (+ 764us) server_negotiation.cc:299] Negotiation successful
0619 18:47:41.826424 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":783}
I20260619 18:47:41.829452 11174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:41.796127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:41477 (local address 127.3.243.65:35367)
0619 18:47:41.797285 (+ 1158us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:41.797300 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:41.798549 (+ 1249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:41.799697 (+ 1148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:41.799711 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:41.803426 (+ 3715us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:47:41.804051 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:41.804064 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:41.805945 (+ 1881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:41.805951 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:41.808288 (+ 2337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:41.808303 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:41.812647 (+ 4344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:41.812655 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:41.812932 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:41.816367 (+ 3435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:47:41.816389 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:47:41.820858 (+ 4469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:47:41.825459 (+ 4601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:41.825463 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:41.825466 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:41.826039 (+ 573us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:41.826513 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:41.826518 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:41.826522 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:41.826723 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:41.827555 (+ 832us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:47:41.827559 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:47:41.827706 (+ 147us) client_negotiation.cc:769] Sending connection context
0619 18:47:41.827832 (+ 126us) client_negotiation.cc:240] Negotiation successful
0619 18:47:41.828043 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1026,"mutex_wait_us":1,"thread_start_us":762,"threads_started":1}
I20260619 18:47:41.829672 11171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:41.793275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.125:37445 (local address 127.3.243.65:57595)
0619 18:47:41.794844 (+ 1569us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:41.794901 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:41.799912 (+ 5011us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:41.800773 (+ 861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:41.800778 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:41.801476 (+ 698us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:47:41.802732 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:41.802762 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:41.809242 (+ 6480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:41.809248 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:41.811320 (+ 2072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:41.811334 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:41.813567 (+ 2233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:41.813577 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:41.813820 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:41.814529 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:47:41.814550 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:47:41.817647 (+ 3097us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:47:41.822742 (+ 5095us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:41.822747 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:41.822750 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:41.823494 (+ 744us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:41.823994 (+ 500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:41.823999 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:41.824003 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:41.824386 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:41.825157 (+ 771us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:47:41.825165 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:47:41.825702 (+ 537us) client_negotiation.cc:769] Sending connection context
0619 18:47:41.827179 (+ 1477us) client_negotiation.cc:240] Negotiation successful
0619 18:47:41.827794 (+ 615us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1178,"mutex_wait_us":29,"spinlock_wait_cycles":24704,"thread_start_us":751,"threads_started":1}
I20260619 18:47:41.830196 11172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:41.793266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.124:32923 (local address 127.3.243.65:45369)
0619 18:47:41.794986 (+ 1720us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:41.795000 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:41.798543 (+ 3543us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:41.799698 (+ 1155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:41.799711 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:41.801281 (+ 1570us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:47:41.802731 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:41.802763 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:41.804567 (+ 1804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:41.804573 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:41.809870 (+ 5297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:41.809889 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:41.811672 (+ 1783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:41.811679 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:41.812035 (+ 356us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:41.813491 (+ 1456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:47:41.813531 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:47:41.817619 (+ 4088us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:47:41.822659 (+ 5040us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:41.822680 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:41.822698 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:41.823568 (+ 870us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:41.824485 (+ 917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:41.824490 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:41.824493 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:41.824704 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:41.825456 (+ 752us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:47:41.825461 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:47:41.825701 (+ 240us) client_negotiation.cc:769] Sending connection context
0619 18:47:41.827147 (+ 1446us) client_negotiation.cc:240] Negotiation successful
0619 18:47:41.829266 (+ 2119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1537,"mutex_wait_us":4092,"spinlock_wait_cycles":143488,"thread_start_us":1029,"threads_started":1}
I20260619 18:47:41.831328 11168 heartbeater.cc:347] Connected to a master server at 127.3.243.125:37445
I20260619 18:47:41.831656 11168 heartbeater.cc:464] Registering TS with master...
I20260619 18:47:41.831749 11173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:41.795618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.65:35367 (local address 127.3.243.126:41477)
0619 18:47:41.796308 (+ 690us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:41.796314 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:41.797399 (+ 1085us) server_negotiation.cc:407] Connection header received
0619 18:47:41.798991 (+ 1592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:41.798999 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:41.799092 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:41.799215 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:41.804278 (+ 5063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:41.805744 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:41.808489 (+ 2745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:41.809394 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:41.809514 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:41.821121 (+ 11607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:41.821129 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:41.821135 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:41.821188 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:41.825293 (+ 4105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:41.826163 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:41.826179 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:41.826182 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:41.826382 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:41.826852 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:41.826857 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:41.826861 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:41.827357 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:41.827498 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:41.830247 (+ 2749us) server_negotiation.cc:299] Negotiation successful
0619 18:47:41.831247 (+ 1000us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":421,"threads_started":1}
I20260619 18:47:41.831863 11167 heartbeater.cc:347] Connected to a master server at 127.3.243.124:32923
I20260619 18:47:41.832020 11167 heartbeater.cc:464] Registering TS with master...
I20260619 18:47:41.832691 11166 heartbeater.cc:347] Connected to a master server at 127.3.243.126:41477
I20260619 18:47:41.832845 11166 heartbeater.cc:464] Registering TS with master...
I20260619 18:47:41.833453 11167 heartbeater.cc:511] Master 127.3.243.124:32923 requested a full tablet report, sending...
I20260619 18:47:41.833681 11166 heartbeater.cc:511] Master 127.3.243.126:41477 requested a full tablet report, sending...
Jun 19 18:47:41 dist-test-slave-m9wg krb5kdc[10155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894857, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.243.66@KRBTEST.COM
I20260619 18:47:41.834537 11168 heartbeater.cc:511] Master 127.3.243.125:37445 requested a full tablet report, sending...
I20260619 18:47:41.835495 10200 ts_manager.cc:194] Re-registered known tserver with Master: 69da4e79b53c43fc8becd20ad510c8b8 (127.3.243.65:34185)
I20260619 18:47:41.836431 10988 ts_manager.cc:194] Registered new tserver with Master: 69da4e79b53c43fc8becd20ad510c8b8 (127.3.243.65:34185)
I20260619 18:47:41.836591 10200 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.243.65@KRBTEST.COM'} at 127.3.243.65:35367
I20260619 18:47:41.838070 10797 ts_manager.cc:194] Registered new tserver with Master: 69da4e79b53c43fc8becd20ad510c8b8 (127.3.243.65:34185)
I20260619 18:47:41.843773 11176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:41.798796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50332 (local address 127.3.243.66:37147)
0619 18:47:41.814517 (+ 15721us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:41.814527 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:41.814559 (+ 32us) server_negotiation.cc:407] Connection header received
0619 18:47:41.814977 (+ 418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:41.814990 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:41.815308 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:41.815482 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:41.818207 (+ 2725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:41.820360 (+ 2153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:41.822742 (+ 2382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:41.828187 (+ 5445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:41.828326 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:41.835576 (+ 7250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:41.835595 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:41.835613 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:41.835658 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:41.840846 (+ 5188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:41.841624 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:41.841630 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:41.841633 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:41.841797 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:41.842157 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:41.842160 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:41.842162 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:41.842514 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:41.842810 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:41.843035 (+ 225us) server_negotiation.cc:299] Negotiation successful
0619 18:47:41.843252 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15478,"thread_start_us":15755,"threads_started":1}
I20260619 18:47:41.845091 10518 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:47:41.926499 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 10395
I20260619 18:47:41.939749 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.66:37147
--local_ip_for_outbound_sockets=127.3.243.66
--tserver_master_addrs=127.3.243.126:41477
--webserver_port=38285
--webserver_interface=127.3.243.66
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/krb5kdc/oryx_127.3.243.66.keytab
--principal=oryx/127.3.243.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:37789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.3.243.126:41477,127.3.243.125:37445
--tserver_master_addrs=127.3.243.126:41477,127.3.243.125:37445,127.3.243.124:32923 with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 18:47:42.056429 10946 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e612a9d44274aadb7de143295bf9e38 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3e612a9d44274aadb7de143295bf9e38" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3e612a9d44274aadb7de143295bf9e38" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 41477 } } peers { permanent_uuid: "ae1e1640d5134c40a77de2ae1fd29530" member_type: VOTER last_known_addr { host: "127.3.243.125" port: 37445 } attrs { promote: false } } peers { permanent_uuid: "ed237dce23ef407fa74b615f8f6c914f" member_type: NON_VOTER last_known_addr { host: "127.3.243.124" port: 32923 } attrs { promote: true } } }
I20260619 18:47:42.056784 10946 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e612a9d44274aadb7de143295bf9e38 [sys.catalog]: This master's current role is: LEADER
I20260619 18:47:42.058223 11180 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 3e612a9d44274aadb7de143295bf9e38: attempting to promote NON_VOTER ed237dce23ef407fa74b615f8f6c914f to VOTER
I20260619 18:47:42.058990 11180 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e612a9d44274aadb7de143295bf9e38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3e612a9d44274aadb7de143295bf9e38" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 41477 } } peers { permanent_uuid: "ae1e1640d5134c40a77de2ae1fd29530" member_type: VOTER last_known_addr { host: "127.3.243.125" port: 37445 } attrs { promote: false } } peers { permanent_uuid: "ed237dce23ef407fa74b615f8f6c914f" member_type: VOTER last_known_addr { host: "127.3.243.124" port: 32923 } attrs { promote: false } }
I20260619 18:47:42.061048 10998 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ed237dce23ef407fa74b615f8f6c914f [term 1 LEARNER]: Refusing update from remote peer 3e612a9d44274aadb7de143295bf9e38: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260619 18:47:42.061075 10807 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ae1e1640d5134c40a77de2ae1fd29530 [term 1 FOLLOWER]: Refusing update from remote peer 3e612a9d44274aadb7de143295bf9e38: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260619 18:47:42.061614 11179 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3e612a9d44274aadb7de143295bf9e38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed237dce23ef407fa74b615f8f6c914f" member_type: VOTER last_known_addr { host: "127.3.243.124" port: 32923 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260619 18:47:42.062029 11180 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3e612a9d44274aadb7de143295bf9e38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae1e1640d5134c40a77de2ae1fd29530" member_type: VOTER last_known_addr { host: "127.3.243.125" port: 37445 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260619 18:47:42.065429 10807 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ae1e1640d5134c40a77de2ae1fd29530 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, ed237dce23ef407fa74b615f8f6c914f (127.3.243.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3e612a9d44274aadb7de143295bf9e38" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 41477 } } peers { permanent_uuid: "ae1e1640d5134c40a77de2ae1fd29530" member_type: VOTER last_known_addr { host: "127.3.243.125" port: 37445 } attrs { promote: false } } peers { permanent_uuid: "ed237dce23ef407fa74b615f8f6c914f" member_type: VOTER last_known_addr { host: "127.3.243.124" port: 32923 } attrs { promote: false } } }
I20260619 18:47:42.066231 10998 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ed237dce23ef407fa74b615f8f6c914f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, ed237dce23ef407fa74b615f8f6c914f (127.3.243.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3e612a9d44274aadb7de143295bf9e38" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 41477 } } peers { permanent_uuid: "ae1e1640d5134c40a77de2ae1fd29530" member_type: VOTER last_known_addr { host: "127.3.243.125" port: 37445 } attrs { promote: false } } peers { permanent_uuid: "ed237dce23ef407fa74b615f8f6c914f" member_type: VOTER last_known_addr { host: "127.3.243.124" port: 32923 } attrs { promote: false } } }
I20260619 18:47:42.064667 11179 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3e612a9d44274aadb7de143295bf9e38 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, ed237dce23ef407fa74b615f8f6c914f (127.3.243.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3e612a9d44274aadb7de143295bf9e38" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 41477 } } peers { permanent_uuid: "ae1e1640d5134c40a77de2ae1fd29530" member_type: VOTER last_known_addr { host: "127.3.243.125" port: 37445 } attrs { promote: false } } peers { permanent_uuid: "ed237dce23ef407fa74b615f8f6c914f" member_type: VOTER last_known_addr { host: "127.3.243.124" port: 32923 } attrs { promote: false } } }
I20260619 18:47:42.072258 11180 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e612a9d44274aadb7de143295bf9e38 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "3e612a9d44274aadb7de143295bf9e38" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3e612a9d44274aadb7de143295bf9e38" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 41477 } } peers { permanent_uuid: "ae1e1640d5134c40a77de2ae1fd29530" member_type: VOTER last_known_addr { host: "127.3.243.125" port: 37445 } attrs { promote: false } } peers { permanent_uuid: "ed237dce23ef407fa74b615f8f6c914f" member_type: VOTER last_known_addr { host: "127.3.243.124" port: 32923 } attrs { promote: false } } }
I20260619 18:47:42.072705 11180 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e612a9d44274aadb7de143295bf9e38 [sys.catalog]: This master's current role is: LEADER
I20260619 18:47:42.073001 11180 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e612a9d44274aadb7de143295bf9e38 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3e612a9d44274aadb7de143295bf9e38" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3e612a9d44274aadb7de143295bf9e38" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 41477 } } peers { permanent_uuid: "ae1e1640d5134c40a77de2ae1fd29530" member_type: VOTER last_known_addr { host: "127.3.243.125" port: 37445 } attrs { promote: false } } peers { permanent_uuid: "ed237dce23ef407fa74b615f8f6c914f" member_type: VOTER last_known_addr { host: "127.3.243.124" port: 32923 } attrs { promote: false } } }
I20260619 18:47:42.073299 11180 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e612a9d44274aadb7de143295bf9e38 [sys.catalog]: This master's current role is: LEADER
I20260619 18:47:42.073563 11180 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e612a9d44274aadb7de143295bf9e38 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3e612a9d44274aadb7de143295bf9e38" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3e612a9d44274aadb7de143295bf9e38" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 41477 } } peers { permanent_uuid: "ae1e1640d5134c40a77de2ae1fd29530" member_type: VOTER last_known_addr { host: "127.3.243.125" port: 37445 } attrs { promote: false } } peers { permanent_uuid: "ed237dce23ef407fa74b615f8f6c914f" member_type: VOTER last_known_addr { host: "127.3.243.124" port: 32923 } attrs { promote: false } } }
I20260619 18:47:42.073822 11180 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e612a9d44274aadb7de143295bf9e38 [sys.catalog]: This master's current role is: LEADER
I20260619 18:47:42.075577 11184 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae1e1640d5134c40a77de2ae1fd29530 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3e612a9d44274aadb7de143295bf9e38" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3e612a9d44274aadb7de143295bf9e38" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 41477 } } peers { permanent_uuid: "ae1e1640d5134c40a77de2ae1fd29530" member_type: VOTER last_known_addr { host: "127.3.243.125" port: 37445 } attrs { promote: false } } peers { permanent_uuid: "ed237dce23ef407fa74b615f8f6c914f" member_type: VOTER last_known_addr { host: "127.3.243.124" port: 32923 } attrs { promote: false } } }
I20260619 18:47:42.076038 11184 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae1e1640d5134c40a77de2ae1fd29530 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:47:42.076449 11183 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed237dce23ef407fa74b615f8f6c914f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3e612a9d44274aadb7de143295bf9e38" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3e612a9d44274aadb7de143295bf9e38" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 41477 } } peers { permanent_uuid: "ae1e1640d5134c40a77de2ae1fd29530" member_type: VOTER last_known_addr { host: "127.3.243.125" port: 37445 } attrs { promote: false } } peers { permanent_uuid: "ed237dce23ef407fa74b615f8f6c914f" member_type: VOTER last_known_addr { host: "127.3.243.124" port: 32923 } attrs { promote: false } } }
I20260619 18:47:42.076798 11183 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed237dce23ef407fa74b615f8f6c914f [sys.catalog]: This master's current role is: FOLLOWER
W20260619 18:47:42.252290 11178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:47:42.252538 11178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:47:42.252637 11178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:47:42.259675 11178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:47:42.259776 11178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:47:42.259893 11178 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.66
I20260619 18:47:42.269122 11178 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:37789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/krb5kdc/oryx_127.3.243.66.keytab
--principal=oryx/127.3.243.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.66:37147
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.243.66
--webserver_port=38285
--tserver_master_addrs=127.3.243.126:41477,127.3.243.125:37445,127.3.243.124:32923
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.66
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:47:42.270256 11178 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:47:42.271485 11178 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:47:42.284333 11190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:42.284375 11192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:42.284410 11189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:47:42.285259 11178 server_base.cc:1034] running on GCE node
I20260619 18:47:42.285811 11178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:47:42.286648 11178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:47:42.287848 11178 hybrid_clock.cc:648] HybridClock initialized: now 1781894862287800 us; error 37 us; skew 500 ppm
I20260619 18:47:42.288128 11178 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:47:42 dist-test-slave-m9wg krb5kdc[10155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894862, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.243.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:47:42.291020 11178 init.cc:377] Logged in from keytab as oryx/127.3.243.66@KRBTEST.COM (short username oryx)
I20260619 18:47:42.292312 11178 webserver.cc:460] Webserver started at http://127.3.243.66:38285/ using document root <none> and password file <none>
I20260619 18:47:42.292940 11178 fs_manager.cc:359] Metadata directory not provided
I20260619 18:47:42.293047 11178 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:47:42.298224 11178 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260619 18:47:42.300612 11199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:42.301821 11178 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 18:47:42.301959 11178 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/ts-1/wal
uuid: "3f5657a7496c4e7b82d4244c01154242"
format_stamp: "Formatted at 2026-06-19 18:47:39 on dist-test-slave-m9wg"
server_key: "97b82f25e1c42724bbfe0d8b34007aec"
server_key_iv: "2b18c224b462f17668012ff6fd2848c0"
server_key_version: "encryptionkey@0"
I20260619 18:47:42.302840 11178 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:47:42.322862 11178 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:47:42.326714 11178 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:47:42.326930 11178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:47:42.327811 11178 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:47:42.329033 11178 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:47:42.329113 11178 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:42.329187 11178 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:47:42.329241 11178 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:42.357147 11178 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.66:37147
I20260619 18:47:42.357187 11311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.66:37147 every 8 connection(s)
I20260619 18:47:42.358534 11178 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/ts-1/data/info.pb
I20260619 18:47:42.367551 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 11178
I20260619 18:47:42.367808 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 10528
Jun 19 18:47:42 dist-test-slave-m9wg krb5kdc[10155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894857, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.243.67@KRBTEST.COM
Jun 19 18:47:42 dist-test-slave-m9wg krb5kdc[10155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894862, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.243.66@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM
Jun 19 18:47:42 dist-test-slave-m9wg krb5kdc[10155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894862, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.243.66@KRBTEST.COM for oryx/127.3.243.125@KRBTEST.COM
Jun 19 18:47:42 dist-test-slave-m9wg krb5kdc[10155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894862, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.243.66@KRBTEST.COM for oryx/127.3.243.124@KRBTEST.COM
I20260619 18:47:42.403883 11321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:42.368989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38760 (local address 127.3.243.67:36847)
0619 18:47:42.372460 (+ 3471us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:42.372470 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:42.372503 (+ 33us) server_negotiation.cc:407] Connection header received
0619 18:47:42.372582 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:42.372604 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:42.372903 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:42.373085 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:42.379649 (+ 6564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:42.381760 (+ 2111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:42.385698 (+ 3938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:42.386826 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:42.386954 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:42.394917 (+ 7963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:42.394930 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:42.394943 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:42.394979 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:42.400102 (+ 5123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:42.400997 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:42.401003 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:42.401005 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:42.401227 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:42.401778 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:42.401782 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:42.401784 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:42.402219 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:42.402597 (+ 378us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:42.402942 (+ 345us) server_negotiation.cc:299] Negotiation successful
0619 18:47:42.403274 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3282,"thread_start_us":3019,"threads_started":1}
I20260619 18:47:42.405622 10651 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:47:42.412874 11323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:42.378365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.66:32877 (local address 127.3.243.126:41477)
0619 18:47:42.383717 (+ 5352us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:42.383723 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:42.385426 (+ 1703us) server_negotiation.cc:407] Connection header received
0619 18:47:42.385603 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:42.385608 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:42.385679 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:42.385777 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:42.388256 (+ 2479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:42.389551 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:42.393216 (+ 3665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:42.394115 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:42.394262 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:42.400134 (+ 5872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:42.400140 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:42.400144 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:42.400239 (+ 95us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:42.404757 (+ 4518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:42.406903 (+ 2146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:42.406910 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:42.406914 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:42.407126 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:42.407748 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:42.407754 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:42.407757 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:42.408217 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:42.408481 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:42.412031 (+ 3550us) server_negotiation.cc:299] Negotiation successful
0619 18:47:42.412353 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5235,"thread_start_us":5134,"threads_started":1}
I20260619 18:47:42.414103 11319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:42.362382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.124:32923 (local address 127.3.243.66:39305)
0619 18:47:42.388864 (+ 26482us) spinlock_profiling.cc:243] Waited 10.2 ms on lock 0x614000009508. stack: 00007f909a7bd66e 00007f909a7bd430 00007f90998445ed 00000000008ea064 00000000008ea02d 00000000008e99fa 00007f90a8664863 00007f90a124877f 00007f90a1246f68 00007f90a127a82f 00007f90a127a58d 00000000009edb7c 00007f909a803344 00007f909a80691b 00007f909a806798 00000000009edb7c
0619 18:47:42.388967 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:42.388987 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:42.389155 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:42.389700 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:42.389705 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:42.390767 (+ 1062us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:47:42.391452 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:42.391465 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:42.393461 (+ 1996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:42.393469 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:42.395553 (+ 2084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:42.395567 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:42.396864 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:42.396869 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:42.397151 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:42.397774 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:47:42.397799 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:47:42.405759 (+ 7960us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:47:42.410511 (+ 4752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:42.410514 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:42.410517 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:42.411134 (+ 617us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:42.411557 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:42.411561 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:42.411563 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:42.411751 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:42.412757 (+ 1006us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:47:42.412761 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:47:42.412926 (+ 165us) client_negotiation.cc:769] Sending connection context
0619 18:47:42.413064 (+ 138us) client_negotiation.cc:240] Negotiation successful
0619 18:47:42.413358 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15262,"mutex_wait_us":62,"spinlock_wait_cycles":23464192,"thread_start_us":15193,"threads_started":1}
I20260619 18:47:42.414925 11322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:42.378318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:41477 (local address 127.3.243.66:32877)
0619 18:47:42.385303 (+ 6985us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:42.385321 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:42.385470 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:42.386018 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:42.386027 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:42.387217 (+ 1190us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:47:42.387788 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:42.387802 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:42.389726 (+ 1924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:42.389733 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:42.393069 (+ 3336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:42.393079 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:42.394558 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:42.394565 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:42.394906 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:42.395844 (+ 938us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:47:42.395895 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:47:42.399885 (+ 3990us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:47:42.404922 (+ 5037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:42.404931 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:42.404947 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:42.405981 (+ 1034us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:42.407292 (+ 1311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:42.407298 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:42.407302 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:42.407624 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:42.410977 (+ 3353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:47:42.410987 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:47:42.411512 (+ 525us) client_negotiation.cc:769] Sending connection context
0619 18:47:42.411916 (+ 404us) client_negotiation.cc:240] Negotiation successful
0619 18:47:42.412539 (+ 623us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6838,"mutex_wait_us":670,"thread_start_us":6641,"threads_started":1}
I20260619 18:47:42.415385 11316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:42.361610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.66:39305 (local address 127.3.243.124:32923)
0619 18:47:42.384511 (+ 22901us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:42.384515 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:42.389070 (+ 4555us) server_negotiation.cc:407] Connection header received
0619 18:47:42.389296 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:42.389303 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:42.389381 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:42.389450 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:42.391647 (+ 2197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:42.393124 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:42.395740 (+ 2616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:42.396744 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:42.396866 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:42.406014 (+ 9148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:42.406021 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:42.406026 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:42.406101 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:42.410391 (+ 4290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:42.411275 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:42.411281 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:42.411285 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:42.411443 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:42.411852 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:42.411856 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:42.411858 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:42.412539 (+ 681us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:42.412685 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:42.414599 (+ 1914us) server_negotiation.cc:299] Negotiation successful
0619 18:47:42.414825 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22803,"thread_start_us":22682,"threads_started":1}
I20260619 18:47:42.416566 11312 heartbeater.cc:347] Connected to a master server at 127.3.243.126:41477
I20260619 18:47:42.416687 11318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:42.362382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.125:37445 (local address 127.3.243.66:40749)
0619 18:47:42.378339 (+ 15957us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:42.378390 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:42.385213 (+ 6823us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:42.386015 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:42.386024 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:42.386876 (+ 852us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:47:42.387696 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:42.387715 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:42.389928 (+ 2213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:42.389933 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:42.392640 (+ 2707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:42.392658 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:42.397211 (+ 4553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:42.397219 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:42.397457 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:42.398100 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:47:42.398122 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:47:42.402109 (+ 3987us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:47:42.408180 (+ 6071us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:42.408185 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:42.408188 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:42.408717 (+ 529us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:42.410931 (+ 2214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:42.410936 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:42.410939 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:42.411195 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:42.414556 (+ 3361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:47:42.414561 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:47:42.414748 (+ 187us) client_negotiation.cc:769] Sending connection context
0619 18:47:42.414888 (+ 140us) client_negotiation.cc:240] Negotiation successful
0619 18:47:42.415888 (+ 1000us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14372,"mutex_wait_us":168,"thread_start_us":14011,"threads_started":1}
I20260619 18:47:42.416818 11317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:42.361653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.66:40749 (local address 127.3.243.125:37445)
0619 18:47:42.377529 (+ 15876us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:42.377535 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:42.378735 (+ 1200us) server_negotiation.cc:407] Connection header received
0619 18:47:42.385493 (+ 6758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:42.385497 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:42.385622 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:42.385701 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:42.387859 (+ 2158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:42.389221 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:42.393350 (+ 4129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:42.394249 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:42.394408 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:42.402356 (+ 7948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:42.402362 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:42.402366 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:42.402459 (+ 93us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:42.407102 (+ 4643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:42.410600 (+ 3498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:42.410608 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:42.410612 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:42.410804 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:42.413969 (+ 3165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:42.413975 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:42.413978 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:42.414431 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:42.414616 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:42.415312 (+ 696us) server_negotiation.cc:299] Negotiation successful
0619 18:47:42.416342 (+ 1030us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15741,"thread_start_us":15867,"threads_started":1}
I20260619 18:47:42.416718 11313 heartbeater.cc:347] Connected to a master server at 127.3.243.124:32923
I20260619 18:47:42.417246 11313 heartbeater.cc:464] Registering TS with master...
I20260619 18:47:42.417817 11314 heartbeater.cc:347] Connected to a master server at 127.3.243.125:37445
I20260619 18:47:42.417963 11314 heartbeater.cc:464] Registering TS with master...
I20260619 18:47:42.416937 11312 heartbeater.cc:464] Registering TS with master...
I20260619 18:47:42.419335 11314 heartbeater.cc:511] Master 127.3.243.125:37445 requested a full tablet report, sending...
I20260619 18:47:42.419373 11313 heartbeater.cc:511] Master 127.3.243.124:32923 requested a full tablet report, sending...
I20260619 18:47:42.420398 11312 heartbeater.cc:511] Master 127.3.243.126:41477 requested a full tablet report, sending...
I20260619 18:47:42.421936 10988 ts_manager.cc:194] Registered new tserver with Master: 3f5657a7496c4e7b82d4244c01154242 (127.3.243.66:37147)
I20260619 18:47:42.423046 10797 ts_manager.cc:194] Registered new tserver with Master: 3f5657a7496c4e7b82d4244c01154242 (127.3.243.66:37147)
I20260619 18:47:42.423444 10200 ts_manager.cc:194] Re-registered known tserver with Master: 3f5657a7496c4e7b82d4244c01154242 (127.3.243.66:37147)
I20260619 18:47:42.424520 10200 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.243.66@KRBTEST.COM'} at 127.3.243.66:32877
I20260619 18:47:42.504357 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 10528
I20260619 18:47:42.518064 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.67:36847
--local_ip_for_outbound_sockets=127.3.243.67
--tserver_master_addrs=127.3.243.126:41477
--webserver_port=37007
--webserver_interface=127.3.243.67
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/krb5kdc/oryx_127.3.243.67.keytab
--principal=oryx/127.3.243.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:37789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.3.243.126:41477,127.3.243.125:37445
--tserver_master_addrs=127.3.243.126:41477,127.3.243.125:37445,127.3.243.124:32923 with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:47:42.825709 11325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:47:42.825986 11325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:47:42.826133 11325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:47:42.833040 11325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:47:42.833151 11325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:47:42.833309 11325 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.67
I20260619 18:47:42.840279 11166 heartbeater.cc:503] Master 127.3.243.126:41477 was elected leader, sending a full tablet report...
I20260619 18:47:42.843710 11325 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:37789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/krb5kdc/oryx_127.3.243.67.keytab
--principal=oryx/127.3.243.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.67:36847
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.243.67
--webserver_port=37007
--tserver_master_addrs=127.3.243.126:41477,127.3.243.125:37445,127.3.243.124:32923
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.67
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:47:42.844955 11325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:47:42.846216 11325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:47:42.858598 11330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:42.858639 11331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:42.858695 11333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:47:42.859634 11325 server_base.cc:1034] running on GCE node
I20260619 18:47:42.860257 11325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:47:42.861141 11325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:47:42.862330 11325 hybrid_clock.cc:648] HybridClock initialized: now 1781894862862290 us; error 32 us; skew 500 ppm
I20260619 18:47:42.862632 11325 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:47:42 dist-test-slave-m9wg krb5kdc[10155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894862, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.243.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:47:42.865569 11325 init.cc:377] Logged in from keytab as oryx/127.3.243.67@KRBTEST.COM (short username oryx)
I20260619 18:47:42.866889 11325 webserver.cc:460] Webserver started at http://127.3.243.67:37007/ using document root <none> and password file <none>
I20260619 18:47:42.867537 11325 fs_manager.cc:359] Metadata directory not provided
I20260619 18:47:42.867679 11325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:47:42.872741 11325 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:47:42.875267 11340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:42.876750 11325 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:47:42.876888 11325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/ts-2/wal
uuid: "7c75efa0e0da41d7883d6fc4bc7c6f89"
format_stamp: "Formatted at 2026-06-19 18:47:39 on dist-test-slave-m9wg"
server_key: "306d575bf5a85c02703afbb83a6c410a"
server_key_iv: "efbca80ac7b9fa5b44605e486e871f72"
server_key_version: "encryptionkey@0"
I20260619 18:47:42.877806 11325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:47:42.901718 11325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:47:42.905840 11325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:47:42.906050 11325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:47:42.906956 11325 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:47:42.908349 11325 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:47:42.908449 11325 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:42.908550 11325 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:47:42.908617 11325 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:42.937718 11325 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.67:36847
I20260619 18:47:42.937773 11452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.67:36847 every 8 connection(s)
I20260619 18:47:42.939196 11325 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/ts-2/data/info.pb
I20260619 18:47:42.944743 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 11325
Jun 19 18:47:42 dist-test-slave-m9wg krb5kdc[10155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894862, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.243.67@KRBTEST.COM for oryx/127.3.243.124@KRBTEST.COM
Jun 19 18:47:42 dist-test-slave-m9wg krb5kdc[10155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894862, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.243.67@KRBTEST.COM for oryx/127.3.243.125@KRBTEST.COM
Jun 19 18:47:42 dist-test-slave-m9wg krb5kdc[10155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894862, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.243.67@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM
I20260619 18:47:42.982316 11457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:42.942197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.67:35153 (local address 127.3.243.125:37445)
0619 18:47:42.957600 (+ 15403us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:42.957607 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:42.959290 (+ 1683us) server_negotiation.cc:407] Connection header received
0619 18:47:42.959532 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:42.959538 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:42.959625 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:42.959718 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:42.962047 (+ 2329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:42.963126 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:42.965277 (+ 2151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:42.965948 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:42.966045 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:42.970422 (+ 4377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:42.970430 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:42.970435 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:42.970513 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:42.975656 (+ 5143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:42.977117 (+ 1461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:42.977122 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:42.977125 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:42.977362 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:42.977930 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:42.977935 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:42.977939 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:42.978367 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:42.978551 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:42.981606 (+ 3055us) server_negotiation.cc:299] Negotiation successful
0619 18:47:42.981820 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15239,"thread_start_us":16147,"threads_started":1}
I20260619 18:47:42.982892 11459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:42.942928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.125:37445 (local address 127.3.243.67:35153)
0619 18:47:42.959174 (+ 16246us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:42.959198 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:42.959388 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:42.959944 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:42.959948 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:42.960570 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:47:42.961129 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:42.961141 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:42.963317 (+ 2176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:42.963323 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:42.965096 (+ 1773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:42.965117 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:42.966328 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:42.966334 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:42.966542 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:42.967263 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:47:42.967288 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:47:42.970142 (+ 2854us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:47:42.975910 (+ 5768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:42.975915 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:42.975919 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:42.976937 (+ 1018us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:42.977507 (+ 570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:42.977512 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:42.977515 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:42.977797 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:42.978629 (+ 832us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:47:42.978638 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:47:42.979144 (+ 506us) client_negotiation.cc:769] Sending connection context
0619 18:47:42.979561 (+ 417us) client_negotiation.cc:240] Negotiation successful
0619 18:47:42.980182 (+ 621us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16108,"mutex_wait_us":266,"thread_start_us":17959,"threads_started":1}
I20260619 18:47:42.984073 11463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:42.958887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:41477 (local address 127.3.243.67:60925)
0619 18:47:42.963523 (+ 4636us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:42.963545 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:42.963654 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:42.964223 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:42.964228 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:42.964797 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:47:42.965132 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:42.965144 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:42.966456 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:42.966464 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:42.968028 (+ 1564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:42.968040 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:42.969115 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:42.969125 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:42.969424 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:42.970142 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:47:42.970172 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:47:42.976500 (+ 6328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:47:42.981391 (+ 4891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:42.981396 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:42.981399 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:42.981854 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:42.982292 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:42.982297 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:42.982300 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:42.982471 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:42.983141 (+ 670us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:47:42.983145 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:47:42.983288 (+ 143us) client_negotiation.cc:769] Sending connection context
0619 18:47:42.983439 (+ 151us) client_negotiation.cc:240] Negotiation successful
0619 18:47:42.983724 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4499,"thread_start_us":1933,"threads_started":1}
I20260619 18:47:42.985061 11464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:42.958977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.67:60925 (local address 127.3.243.126:41477)
0619 18:47:42.962565 (+ 3588us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:42.962570 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:42.963782 (+ 1212us) server_negotiation.cc:407] Connection header received
0619 18:47:42.963849 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:42.963855 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:42.963934 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:42.964030 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:42.965303 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:42.966293 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:42.968191 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:42.968569 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:42.968676 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:42.976845 (+ 8169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:42.976852 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:42.976857 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:42.976899 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:42.981245 (+ 4346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:42.981995 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:42.982001 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:42.982004 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:42.982160 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:42.982602 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:42.982607 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:42.982611 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:42.982954 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:42.983084 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:42.984494 (+ 1410us) server_negotiation.cc:299] Negotiation successful
0619 18:47:42.984730 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3481,"thread_start_us":3378,"threads_started":1}
I20260619 18:47:42.985360 11455 heartbeater.cc:347] Connected to a master server at 127.3.243.125:37445
I20260619 18:47:42.985697 11455 heartbeater.cc:464] Registering TS with master...
I20260619 18:47:42.985806 11453 heartbeater.cc:347] Connected to a master server at 127.3.243.126:41477
I20260619 18:47:42.985926 11453 heartbeater.cc:464] Registering TS with master...
I20260619 18:47:42.986129 11458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:42.942220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.67:47863 (local address 127.3.243.124:32923)
0619 18:47:42.957951 (+ 15731us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:42.957955 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:42.957973 (+ 18us) server_negotiation.cc:407] Connection header received
0619 18:47:42.958029 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:42.958033 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:42.958111 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:42.958225 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:42.960957 (+ 2732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:42.961871 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:42.964271 (+ 2400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:42.964870 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:42.964972 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:42.970796 (+ 5824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:42.970803 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:42.970809 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:42.970851 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:42.974994 (+ 4143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:42.983946 (+ 8952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:42.983951 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:42.983954 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:42.984104 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:42.984581 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:42.984586 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:42.984589 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:42.984929 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:42.985058 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:42.985581 (+ 523us) server_negotiation.cc:299] Negotiation successful
0619 18:47:42.985808 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15629,"thread_start_us":15525,"threads_started":1}
I20260619 18:47:42.986615 11453 heartbeater.cc:511] Master 127.3.243.126:41477 requested a full tablet report, sending...
I20260619 18:47:42.987043 11460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:42.942899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.124:32923 (local address 127.3.243.67:47863)
0619 18:47:42.954393 (+ 11494us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:42.954463 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:42.957166 (+ 2703us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:42.958586 (+ 1420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:42.958601 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:42.959728 (+ 1127us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:47:42.960762 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:42.960792 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:42.962037 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:42.962042 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:42.964090 (+ 2048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:42.964103 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:42.965323 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:42.965330 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:42.965597 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:42.966360 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:47:42.966390 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:47:42.970534 (+ 4144us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:47:42.975168 (+ 4634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:42.975177 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:42.975197 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:42.983799 (+ 8602us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:42.984296 (+ 497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:42.984300 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:42.984304 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:42.984455 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:42.985138 (+ 683us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:47:42.985142 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:47:42.985309 (+ 167us) client_negotiation.cc:769] Sending connection context
0619 18:47:42.986365 (+ 1056us) client_negotiation.cc:240] Negotiation successful
0619 18:47:42.986651 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11043,"mutex_wait_us":2,"spinlock_wait_cycles":34688,"thread_start_us":14640,"threads_started":1}
I20260619 18:47:42.987587 11455 heartbeater.cc:511] Master 127.3.243.125:37445 requested a full tablet report, sending...
I20260619 18:47:42.988816 11454 heartbeater.cc:347] Connected to a master server at 127.3.243.124:32923
I20260619 18:47:42.988926 11454 heartbeater.cc:464] Registering TS with master...
I20260619 18:47:42.988902 10797 ts_manager.cc:194] Registered new tserver with Master: 7c75efa0e0da41d7883d6fc4bc7c6f89 (127.3.243.67:36847)
I20260619 18:47:42.988909 10200 ts_manager.cc:194] Re-registered known tserver with Master: 7c75efa0e0da41d7883d6fc4bc7c6f89 (127.3.243.67:36847)
I20260619 18:47:42.989307 11454 heartbeater.cc:511] Master 127.3.243.124:32923 requested a full tablet report, sending...
I20260619 18:47:42.989964 10200 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.243.67@KRBTEST.COM'} at 127.3.243.67:60925
I20260619 18:47:42.990348 10988 ts_manager.cc:194] Registered new tserver with Master: 7c75efa0e0da41d7883d6fc4bc7c6f89 (127.3.243.67:36847)
I20260619 18:47:43.005250 11468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:42.972044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33032 (local address 127.3.243.126:41477)
0619 18:47:42.983854 (+ 11810us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:42.983860 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:42.987715 (+ 3855us) server_negotiation.cc:407] Connection header received
0619 18:47:42.987777 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:42.987783 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:42.987863 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:42.987979 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:42.992441 (+ 4462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:42.993462 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:42.996236 (+ 2774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:42.996717 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:42.996831 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:42.999897 (+ 3066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:42.999902 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:42.999905 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:42.999935 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:43.002368 (+ 2433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:43.003103 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:43.003108 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:43.003110 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:43.003237 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:43.003641 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:43.003647 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:43.003650 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:43.003987 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:43.004266 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:43.004551 (+ 285us) server_negotiation.cc:299] Negotiation successful
0619 18:47:43.004942 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11682,"thread_start_us":11818,"threads_started":1}
I20260619 18:47:43.007965 11457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:42.978742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45696 (local address 127.3.243.125:37445)
0619 18:47:42.982550 (+ 3808us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:42.982556 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:42.992974 (+ 10418us) server_negotiation.cc:407] Connection header received
0619 18:47:42.993116 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:42.993120 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:42.993230 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:42.993317 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:42.995879 (+ 2562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:42.996864 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:42.998543 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:42.999535 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:42.999645 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:43.002473 (+ 2828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:43.002479 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:43.002484 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:43.002528 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:43.005178 (+ 2650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:43.005885 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:43.005891 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:43.005895 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:43.006045 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:43.006384 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:43.006390 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:43.006393 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:43.006710 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:43.006942 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:43.007319 (+ 377us) server_negotiation.cc:299] Negotiation successful
0619 18:47:43.007620 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3711,"thread_start_us":5297,"threads_started":1}
I20260619 18:47:43.009766 11458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:42.987519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39932 (local address 127.3.243.124:32923)
0619 18:47:42.987679 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:42.987684 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:42.993208 (+ 5524us) server_negotiation.cc:407] Connection header received
0619 18:47:42.993317 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:42.993322 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:42.993417 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:42.993519 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:42.995523 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:42.996340 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:42.998139 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:42.998841 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:42.998996 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:43.002408 (+ 3412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:43.002414 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:43.002418 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:43.002488 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:43.006589 (+ 4101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:43.007583 (+ 994us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:43.007588 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:43.007592 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:43.007753 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:43.008188 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:43.008193 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:43.008195 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:43.008517 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:43.008745 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:43.009329 (+ 584us) server_negotiation.cc:299] Negotiation successful
0619 18:47:43.009557 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 18:47:43.010084 10200 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33032:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:47:43.016698 10200 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:47:43.049716 11481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:43.040714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50334 (local address 127.3.243.66:37147)
0619 18:47:43.042177 (+ 1463us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:43.042186 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:43.042219 (+ 33us) server_negotiation.cc:407] Connection header received
0619 18:47:43.042330 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:43.042339 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:43.042658 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:43.042882 (+ 224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:47:43.044102 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:43.045552 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:43.047774 (+ 2222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:43.048729 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:43.048852 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:43.049069 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:43.049301 (+ 232us) server_negotiation.cc:299] Negotiation successful
0619 18:47:43.049448 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1321,"thread_start_us":1215,"threads_started":1}
I20260619 18:47:43.049777 11482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:43.040425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.66:37147 (local address 127.0.0.1:50334)
0619 18:47:43.041430 (+ 1005us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:43.041451 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:43.041616 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:43.043112 (+ 1496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:43.043117 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:43.043145 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:47:43.043836 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:43.043852 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:43.045675 (+ 1823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:43.045681 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:43.047631 (+ 1950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:43.047643 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:43.048906 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:43.048912 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:43.049121 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:43.049160 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 18:47:43.049248 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 18:47:43.049372 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":870,"mutex_wait_us":67,"thread_start_us":2849,"threads_started":1}
I20260619 18:47:43.052498 11485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:43.042541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38766 (local address 127.3.243.67:36847)
0619 18:47:43.043793 (+ 1252us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:43.043801 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:43.043934 (+ 133us) server_negotiation.cc:407] Connection header received
0619 18:47:43.044105 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:43.044112 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:43.044357 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:43.044539 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:47:43.045112 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:43.046542 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:43.048991 (+ 2449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:43.051470 (+ 2479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:43.051600 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:43.051787 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:43.051982 (+ 195us) server_negotiation.cc:299] Negotiation successful
0619 18:47:43.052125 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":357,"threads_started":1}
I20260619 18:47:43.052922 11480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:43.041962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.67:36847 (local address 127.0.0.1:38766)
0619 18:47:43.043830 (+ 1868us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:43.043846 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:43.043959 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:43.044553 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:43.044558 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:43.044587 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:47:43.044919 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:43.044929 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:43.046695 (+ 1766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:43.046700 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:43.048816 (+ 2116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:43.048829 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:43.051639 (+ 2810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:43.051645 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:43.051808 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:43.051840 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 18:47:43.051936 (+ 96us) client_negotiation.cc:240] Negotiation successful
0619 18:47:43.052047 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1556,"mutex_wait_us":25,"thread_start_us":386,"threads_started":1}
I20260619 18:47:43.054098 11484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:43.040783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.65:34185 (local address 127.0.0.1:50180)
0619 18:47:43.042492 (+ 1709us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:43.042515 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:43.042667 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:43.046486 (+ 3819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:43.046491 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:43.046516 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:47:43.046979 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:43.046990 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:43.048965 (+ 1975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:43.048970 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:43.050951 (+ 1981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:43.050965 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:43.053338 (+ 2373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:43.053344 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:43.053500 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:43.053529 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 18:47:43.053615 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 18:47:43.053740 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1602,"mutex_wait_us":3,"thread_start_us":702,"threads_started":1}
I20260619 18:47:43.054391 11483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:43.041003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50180 (local address 127.3.243.65:34185)
0619 18:47:43.045964 (+ 4961us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:43.045972 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:43.045997 (+ 25us) server_negotiation.cc:407] Connection header received
0619 18:47:43.046087 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:43.046094 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:43.046291 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:43.047175 (+ 884us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:47:43.047383 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:43.048818 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:43.051169 (+ 2351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:43.052469 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:43.053278 (+ 809us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:43.053462 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:43.053853 (+ 391us) server_negotiation.cc:299] Negotiation successful
0619 18:47:43.054003 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4806,"thread_start_us":4705,"threads_started":1}
I20260619 18:47:43.053970 11247 tablet_service.cc:1467] Processing CreateTablet for tablet 4187a78e6c9149fa9127b1ba98f33d2d (DEFAULT_TABLE table=test-table [id=d8926a32c03245168a288f2e22c1ce01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:47:43.055565 11247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4187a78e6c9149fa9127b1ba98f33d2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:47:43.057179 11388 tablet_service.cc:1467] Processing CreateTablet for tablet 4187a78e6c9149fa9127b1ba98f33d2d (DEFAULT_TABLE table=test-table [id=d8926a32c03245168a288f2e22c1ce01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:47:43.057564 11101 tablet_service.cc:1467] Processing CreateTablet for tablet 4187a78e6c9149fa9127b1ba98f33d2d (DEFAULT_TABLE table=test-table [id=d8926a32c03245168a288f2e22c1ce01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:47:43.058506 11388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4187a78e6c9149fa9127b1ba98f33d2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:47:43.058748 11101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4187a78e6c9149fa9127b1ba98f33d2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:47:43.068962 11486 tablet_bootstrap.cc:492] T 4187a78e6c9149fa9127b1ba98f33d2d P 3f5657a7496c4e7b82d4244c01154242: Bootstrap starting.
I20260619 18:47:43.073196 11486 tablet_bootstrap.cc:654] T 4187a78e6c9149fa9127b1ba98f33d2d P 3f5657a7496c4e7b82d4244c01154242: Neither blocks nor log segments found. Creating new log.
I20260619 18:47:43.074390 11486 log.cc:824] T 4187a78e6c9149fa9127b1ba98f33d2d P 3f5657a7496c4e7b82d4244c01154242: Log is configured to *not* fsync() on all Append() calls
I20260619 18:47:43.075922 11487 tablet_bootstrap.cc:492] T 4187a78e6c9149fa9127b1ba98f33d2d P 7c75efa0e0da41d7883d6fc4bc7c6f89: Bootstrap starting.
I20260619 18:47:43.078076 11488 tablet_bootstrap.cc:492] T 4187a78e6c9149fa9127b1ba98f33d2d P 69da4e79b53c43fc8becd20ad510c8b8: Bootstrap starting.
I20260619 18:47:43.080026 11487 tablet_bootstrap.cc:654] T 4187a78e6c9149fa9127b1ba98f33d2d P 7c75efa0e0da41d7883d6fc4bc7c6f89: Neither blocks nor log segments found. Creating new log.
I20260619 18:47:43.080849 11488 tablet_bootstrap.cc:654] T 4187a78e6c9149fa9127b1ba98f33d2d P 69da4e79b53c43fc8becd20ad510c8b8: Neither blocks nor log segments found. Creating new log.
I20260619 18:47:43.081216 11487 log.cc:824] T 4187a78e6c9149fa9127b1ba98f33d2d P 7c75efa0e0da41d7883d6fc4bc7c6f89: Log is configured to *not* fsync() on all Append() calls
I20260619 18:47:43.082345 11488 log.cc:824] T 4187a78e6c9149fa9127b1ba98f33d2d P 69da4e79b53c43fc8becd20ad510c8b8: Log is configured to *not* fsync() on all Append() calls
I20260619 18:47:43.082816 11486 tablet_bootstrap.cc:492] T 4187a78e6c9149fa9127b1ba98f33d2d P 3f5657a7496c4e7b82d4244c01154242: No bootstrap required, opened a new log
I20260619 18:47:43.083086 11486 ts_tablet_manager.cc:1397] T 4187a78e6c9149fa9127b1ba98f33d2d P 3f5657a7496c4e7b82d4244c01154242: Time spent bootstrapping tablet: real 0.014s user 0.012s sys 0.000s
I20260619 18:47:43.084559 11487 tablet_bootstrap.cc:492] T 4187a78e6c9149fa9127b1ba98f33d2d P 7c75efa0e0da41d7883d6fc4bc7c6f89: No bootstrap required, opened a new log
I20260619 18:47:43.084836 11487 ts_tablet_manager.cc:1397] T 4187a78e6c9149fa9127b1ba98f33d2d P 7c75efa0e0da41d7883d6fc4bc7c6f89: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260619 18:47:43.086048 11488 tablet_bootstrap.cc:492] T 4187a78e6c9149fa9127b1ba98f33d2d P 69da4e79b53c43fc8becd20ad510c8b8: No bootstrap required, opened a new log
I20260619 18:47:43.086275 11488 ts_tablet_manager.cc:1397] T 4187a78e6c9149fa9127b1ba98f33d2d P 69da4e79b53c43fc8becd20ad510c8b8: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260619 18:47:43.095144 11486 raft_consensus.cc:348] T 4187a78e6c9149fa9127b1ba98f33d2d P 3f5657a7496c4e7b82d4244c01154242 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f5657a7496c4e7b82d4244c01154242" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 37147 } } peers { permanent_uuid: "69da4e79b53c43fc8becd20ad510c8b8" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 34185 } } peers { permanent_uuid: "7c75efa0e0da41d7883d6fc4bc7c6f89" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 36847 } }
I20260619 18:47:43.095764 11486 raft_consensus.cc:374] T 4187a78e6c9149fa9127b1ba98f33d2d P 3f5657a7496c4e7b82d4244c01154242 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:47:43.095922 11486 raft_consensus.cc:729] T 4187a78e6c9149fa9127b1ba98f33d2d P 3f5657a7496c4e7b82d4244c01154242 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f5657a7496c4e7b82d4244c01154242, State: Initialized, Role: FOLLOWER
I20260619 18:47:43.096633 11488 raft_consensus.cc:348] T 4187a78e6c9149fa9127b1ba98f33d2d P 69da4e79b53c43fc8becd20ad510c8b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f5657a7496c4e7b82d4244c01154242" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 37147 } } peers { permanent_uuid: "69da4e79b53c43fc8becd20ad510c8b8" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 34185 } } peers { permanent_uuid: "7c75efa0e0da41d7883d6fc4bc7c6f89" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 36847 } }
I20260619 18:47:43.096714 11486 consensus_queue.cc:260] T 4187a78e6c9149fa9127b1ba98f33d2d P 3f5657a7496c4e7b82d4244c01154242 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f5657a7496c4e7b82d4244c01154242" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 37147 } } peers { permanent_uuid: "69da4e79b53c43fc8becd20ad510c8b8" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 34185 } } peers { permanent_uuid: "7c75efa0e0da41d7883d6fc4bc7c6f89" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 36847 } }
I20260619 18:47:43.096642 11487 raft_consensus.cc:348] T 4187a78e6c9149fa9127b1ba98f33d2d P 7c75efa0e0da41d7883d6fc4bc7c6f89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f5657a7496c4e7b82d4244c01154242" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 37147 } } peers { permanent_uuid: "69da4e79b53c43fc8becd20ad510c8b8" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 34185 } } peers { permanent_uuid: "7c75efa0e0da41d7883d6fc4bc7c6f89" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 36847 } }
I20260619 18:47:43.097081 11488 raft_consensus.cc:374] T 4187a78e6c9149fa9127b1ba98f33d2d P 69da4e79b53c43fc8becd20ad510c8b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:47:43.097168 11487 raft_consensus.cc:374] T 4187a78e6c9149fa9127b1ba98f33d2d P 7c75efa0e0da41d7883d6fc4bc7c6f89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:47:43.097244 11488 raft_consensus.cc:729] T 4187a78e6c9149fa9127b1ba98f33d2d P 69da4e79b53c43fc8becd20ad510c8b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69da4e79b53c43fc8becd20ad510c8b8, State: Initialized, Role: FOLLOWER
I20260619 18:47:43.097313 11487 raft_consensus.cc:729] T 4187a78e6c9149fa9127b1ba98f33d2d P 7c75efa0e0da41d7883d6fc4bc7c6f89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c75efa0e0da41d7883d6fc4bc7c6f89, State: Initialized, Role: FOLLOWER
I20260619 18:47:43.097863 11488 consensus_queue.cc:260] T 4187a78e6c9149fa9127b1ba98f33d2d P 69da4e79b53c43fc8becd20ad510c8b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f5657a7496c4e7b82d4244c01154242" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 37147 } } peers { permanent_uuid: "69da4e79b53c43fc8becd20ad510c8b8" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 34185 } } peers { permanent_uuid: "7c75efa0e0da41d7883d6fc4bc7c6f89" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 36847 } }
I20260619 18:47:43.098070 11487 consensus_queue.cc:260] T 4187a78e6c9149fa9127b1ba98f33d2d P 7c75efa0e0da41d7883d6fc4bc7c6f89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f5657a7496c4e7b82d4244c01154242" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 37147 } } peers { permanent_uuid: "69da4e79b53c43fc8becd20ad510c8b8" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 34185 } } peers { permanent_uuid: "7c75efa0e0da41d7883d6fc4bc7c6f89" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 36847 } }
I20260619 18:47:43.100287 11488 ts_tablet_manager.cc:1428] T 4187a78e6c9149fa9127b1ba98f33d2d P 69da4e79b53c43fc8becd20ad510c8b8: Time spent starting tablet: real 0.014s user 0.009s sys 0.006s
I20260619 18:47:43.107278 11312 heartbeater.cc:503] Master 127.3.243.126:41477 was elected leader, sending a full tablet report...
I20260619 18:47:43.108397 11453 heartbeater.cc:503] Master 127.3.243.126:41477 was elected leader, sending a full tablet report...
I20260619 18:47:43.109419 11487 ts_tablet_manager.cc:1428] T 4187a78e6c9149fa9127b1ba98f33d2d P 7c75efa0e0da41d7883d6fc4bc7c6f89: Time spent starting tablet: real 0.024s user 0.009s sys 0.011s
I20260619 18:47:43.110011 11486 ts_tablet_manager.cc:1428] T 4187a78e6c9149fa9127b1ba98f33d2d P 3f5657a7496c4e7b82d4244c01154242: Time spent starting tablet: real 0.027s user 0.014s sys 0.008s
W20260619 18:47:43.118671 11315 tablet.cc:2369] T 4187a78e6c9149fa9127b1ba98f33d2d P 3f5657a7496c4e7b82d4244c01154242: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 18:47:43.192263 11456 tablet.cc:2369] T 4187a78e6c9149fa9127b1ba98f33d2d P 7c75efa0e0da41d7883d6fc4bc7c6f89: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:47:43.235827 11493 raft_consensus.cc:482] T 4187a78e6c9149fa9127b1ba98f33d2d P 69da4e79b53c43fc8becd20ad510c8b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:47:43.236111 11493 raft_consensus.cc:504] T 4187a78e6c9149fa9127b1ba98f33d2d P 69da4e79b53c43fc8becd20ad510c8b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f5657a7496c4e7b82d4244c01154242" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 37147 } } peers { permanent_uuid: "69da4e79b53c43fc8becd20ad510c8b8" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 34185 } } peers { permanent_uuid: "7c75efa0e0da41d7883d6fc4bc7c6f89" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 36847 } }
I20260619 18:47:43.237811 11493 leader_election.cc:288] T 4187a78e6c9149fa9127b1ba98f33d2d P 69da4e79b53c43fc8becd20ad510c8b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f5657a7496c4e7b82d4244c01154242 (127.3.243.66:37147), 7c75efa0e0da41d7883d6fc4bc7c6f89 (127.3.243.67:36847)
I20260619 18:47:43.246127 11485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:43.238329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.65:41203 (local address 127.3.243.67:36847)
0619 18:47:43.238600 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:43.238607 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:43.239146 (+ 539us) server_negotiation.cc:407] Connection header received
0619 18:47:43.239300 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:43.239304 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:43.239362 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:43.239433 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:47:43.240405 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:43.241079 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:43.243893 (+ 2814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:43.245082 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:43.245249 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:43.245322 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:43.245758 (+ 436us) server_negotiation.cc:299] Negotiation successful
0619 18:47:43.245886 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122}
I20260619 18:47:43.246174 11496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:43.238050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.67:36847 (local address 127.3.243.65:41203)
0619 18:47:43.239037 (+ 987us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:43.239055 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:43.239175 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:43.239458 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:43.239462 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:43.239484 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:47:43.240270 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:43.240279 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:43.241210 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:43.241215 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:43.243702 (+ 2487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:43.243716 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:43.245314 (+ 1598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:43.245321 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:43.245531 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:43.245609 (+ 78us) client_negotiation.cc:769] Sending connection context
0619 18:47:43.245701 (+ 92us) client_negotiation.cc:240] Negotiation successful
0619 18:47:43.245884 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":879,"mutex_wait_us":154,"thread_start_us":738,"threads_started":1}
I20260619 18:47:43.246186 11481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:43.238195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.65:47805 (local address 127.3.243.66:37147)
0619 18:47:43.238416 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:43.238422 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:43.239069 (+ 647us) server_negotiation.cc:407] Connection header received
0619 18:47:43.239253 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:43.239257 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:43.239317 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:43.239387 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:47:43.240346 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:43.241106 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:43.243853 (+ 2747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:43.245145 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:43.245285 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:43.245346 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:43.245855 (+ 509us) server_negotiation.cc:299] Negotiation successful
0619 18:47:43.245974 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 18:47:43.246734 11495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:43.237993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.66:37147 (local address 127.3.243.65:47805)
0619 18:47:43.238967 (+ 974us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:43.238983 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:43.239121 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:43.239428 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:43.239433 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:43.239454 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:47:43.240216 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:43.240227 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:43.241240 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:43.241244 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:43.243683 (+ 2439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:43.243698 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:43.245526 (+ 1828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:43.245532 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:43.245684 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:43.245715 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 18:47:43.246380 (+ 665us) client_negotiation.cc:240] Negotiation successful
0619 18:47:43.246468 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":845,"mutex_wait_us":128,"thread_start_us":596,"threads_started":1}
I20260619 18:47:43.247140 11408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4187a78e6c9149fa9127b1ba98f33d2d" candidate_uuid: "69da4e79b53c43fc8becd20ad510c8b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c75efa0e0da41d7883d6fc4bc7c6f89" is_pre_election: true
I20260619 18:47:43.247570 11408 raft_consensus.cc:2454] T 4187a78e6c9149fa9127b1ba98f33d2d P 7c75efa0e0da41d7883d6fc4bc7c6f89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69da4e79b53c43fc8becd20ad510c8b8 in term 0.
I20260619 18:47:43.248129 11057 leader_election.cc:302] T 4187a78e6c9149fa9127b1ba98f33d2d P 69da4e79b53c43fc8becd20ad510c8b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69da4e79b53c43fc8becd20ad510c8b8, 7c75efa0e0da41d7883d6fc4bc7c6f89; no voters:
I20260619 18:47:43.248517 11493 raft_consensus.cc:2781] T 4187a78e6c9149fa9127b1ba98f33d2d P 69da4e79b53c43fc8becd20ad510c8b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:47:43.248668 11493 raft_consensus.cc:482] T 4187a78e6c9149fa9127b1ba98f33d2d P 69da4e79b53c43fc8becd20ad510c8b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:47:43.248782 11493 raft_consensus.cc:3037] T 4187a78e6c9149fa9127b1ba98f33d2d P 69da4e79b53c43fc8becd20ad510c8b8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:47:43.248704 11267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4187a78e6c9149fa9127b1ba98f33d2d" candidate_uuid: "69da4e79b53c43fc8becd20ad510c8b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f5657a7496c4e7b82d4244c01154242" is_pre_election: true
I20260619 18:47:43.249233 11267 raft_consensus.cc:2454] T 4187a78e6c9149fa9127b1ba98f33d2d P 3f5657a7496c4e7b82d4244c01154242 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69da4e79b53c43fc8becd20ad510c8b8 in term 0.
I20260619 18:47:43.251194 11493 raft_consensus.cc:504] T 4187a78e6c9149fa9127b1ba98f33d2d P 69da4e79b53c43fc8becd20ad510c8b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f5657a7496c4e7b82d4244c01154242" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 37147 } } peers { permanent_uuid: "69da4e79b53c43fc8becd20ad510c8b8" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 34185 } } peers { permanent_uuid: "7c75efa0e0da41d7883d6fc4bc7c6f89" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 36847 } }
I20260619 18:47:43.251771 11493 leader_election.cc:288] T 4187a78e6c9149fa9127b1ba98f33d2d P 69da4e79b53c43fc8becd20ad510c8b8 [CANDIDATE]: Term 1 election: Requested vote from peers 3f5657a7496c4e7b82d4244c01154242 (127.3.243.66:37147), 7c75efa0e0da41d7883d6fc4bc7c6f89 (127.3.243.67:36847)
I20260619 18:47:43.252082 11267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4187a78e6c9149fa9127b1ba98f33d2d" candidate_uuid: "69da4e79b53c43fc8becd20ad510c8b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f5657a7496c4e7b82d4244c01154242"
I20260619 18:47:43.252264 11267 raft_consensus.cc:3037] T 4187a78e6c9149fa9127b1ba98f33d2d P 3f5657a7496c4e7b82d4244c01154242 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:47:43.252646 11408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4187a78e6c9149fa9127b1ba98f33d2d" candidate_uuid: "69da4e79b53c43fc8becd20ad510c8b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c75efa0e0da41d7883d6fc4bc7c6f89"
I20260619 18:47:43.252816 11408 raft_consensus.cc:3037] T 4187a78e6c9149fa9127b1ba98f33d2d P 7c75efa0e0da41d7883d6fc4bc7c6f89 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:47:43.254520 11267 raft_consensus.cc:2454] T 4187a78e6c9149fa9127b1ba98f33d2d P 3f5657a7496c4e7b82d4244c01154242 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69da4e79b53c43fc8becd20ad510c8b8 in term 1.
I20260619 18:47:43.254861 11056 leader_election.cc:302] T 4187a78e6c9149fa9127b1ba98f33d2d P 69da4e79b53c43fc8becd20ad510c8b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f5657a7496c4e7b82d4244c01154242, 69da4e79b53c43fc8becd20ad510c8b8; no voters:
I20260619 18:47:43.255024 11408 raft_consensus.cc:2454] T 4187a78e6c9149fa9127b1ba98f33d2d P 7c75efa0e0da41d7883d6fc4bc7c6f89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69da4e79b53c43fc8becd20ad510c8b8 in term 1.
I20260619 18:47:43.255118 11493 raft_consensus.cc:2781] T 4187a78e6c9149fa9127b1ba98f33d2d P 69da4e79b53c43fc8becd20ad510c8b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:47:43.255565 11493 raft_consensus.cc:686] T 4187a78e6c9149fa9127b1ba98f33d2d P 69da4e79b53c43fc8becd20ad510c8b8 [term 1 LEADER]: Becoming Leader. State: Replica: 69da4e79b53c43fc8becd20ad510c8b8, State: Running, Role: LEADER
I20260619 18:47:43.256080 11493 consensus_queue.cc:237] T 4187a78e6c9149fa9127b1ba98f33d2d P 69da4e79b53c43fc8becd20ad510c8b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f5657a7496c4e7b82d4244c01154242" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 37147 } } peers { permanent_uuid: "69da4e79b53c43fc8becd20ad510c8b8" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 34185 } } peers { permanent_uuid: "7c75efa0e0da41d7883d6fc4bc7c6f89" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 36847 } }
I20260619 18:47:43.261073 10200 catalog_manager.cc:5495] T 4187a78e6c9149fa9127b1ba98f33d2d P 69da4e79b53c43fc8becd20ad510c8b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 69da4e79b53c43fc8becd20ad510c8b8 (127.3.243.65). New cstate: current_term: 1 leader_uuid: "69da4e79b53c43fc8becd20ad510c8b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f5657a7496c4e7b82d4244c01154242" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 37147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69da4e79b53c43fc8becd20ad510c8b8" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 34185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7c75efa0e0da41d7883d6fc4bc7c6f89" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 36847 } health_report { overall_health: UNKNOWN } } }
I20260619 18:47:43.284487 11483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:43.279024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50194 (local address 127.3.243.65:34185)
0619 18:47:43.279230 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:43.279235 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:43.279317 (+ 82us) server_negotiation.cc:407] Connection header received
0619 18:47:43.279473 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:43.279477 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:43.279554 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:43.279694 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:47:43.280036 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:43.280893 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:43.282233 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:43.282610 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:43.282701 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:43.283466 (+ 765us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:47:43.283942 (+ 476us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:47:43.284044 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:43.284127 (+ 83us) server_negotiation.cc:299] Negotiation successful
0619 18:47:43.284243 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 18:47:43.291709 11267 raft_consensus.cc:1261] T 4187a78e6c9149fa9127b1ba98f33d2d P 3f5657a7496c4e7b82d4244c01154242 [term 1 FOLLOWER]: Refusing update from remote peer 69da4e79b53c43fc8becd20ad510c8b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:47:43.291712 11408 raft_consensus.cc:1261] T 4187a78e6c9149fa9127b1ba98f33d2d P 7c75efa0e0da41d7883d6fc4bc7c6f89 [term 1 FOLLOWER]: Refusing update from remote peer 69da4e79b53c43fc8becd20ad510c8b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:47:43.292623 11497 consensus_queue.cc:1035] T 4187a78e6c9149fa9127b1ba98f33d2d P 69da4e79b53c43fc8becd20ad510c8b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c75efa0e0da41d7883d6fc4bc7c6f89" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 36847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:47:43.292954 11493 consensus_queue.cc:1035] T 4187a78e6c9149fa9127b1ba98f33d2d P 69da4e79b53c43fc8becd20ad510c8b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f5657a7496c4e7b82d4244c01154242" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 37147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 18:47:43.293648 11169 tablet.cc:2369] T 4187a78e6c9149fa9127b1ba98f33d2d P 69da4e79b53c43fc8becd20ad510c8b8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:47:43.307576 11500 mvcc.cc:205] Tried to move back new op lower bound from 7298641360034398208 to 7298641359900688384. Current Snapshot: MvccSnapshot[applied={T|T < 7298641360034398208}]
I20260619 18:47:43.313378 11501 mvcc.cc:205] Tried to move back new op lower bound from 7298641360034398208 to 7298641359900688384. Current Snapshot: MvccSnapshot[applied={T|T < 7298641360034398208}]
I20260619 18:47:43.334358 10200 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33032:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:47:43.334578 10200 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33032:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:47:43.339998 10200 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3e612a9d44274aadb7de143295bf9e38: Sending DeleteTablet for 3 replicas of tablet 4187a78e6c9149fa9127b1ba98f33d2d
I20260619 18:47:43.341411 11247 tablet_service.cc:1514] Processing DeleteTablet for tablet 4187a78e6c9149fa9127b1ba98f33d2d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:47:43 UTC) from {username='oryx', principal='oryx/127.3.243.126@KRBTEST.COM'} at 127.0.0.1:50334
I20260619 18:47:43.341431 11101 tablet_service.cc:1514] Processing DeleteTablet for tablet 4187a78e6c9149fa9127b1ba98f33d2d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:47:43 UTC) from {username='oryx', principal='oryx/127.3.243.126@KRBTEST.COM'} at 127.0.0.1:50180
I20260619 18:47:43.343098 11388 tablet_service.cc:1514] Processing DeleteTablet for tablet 4187a78e6c9149fa9127b1ba98f33d2d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:47:43 UTC) from {username='oryx', principal='oryx/127.3.243.126@KRBTEST.COM'} at 127.0.0.1:38766
I20260619 18:47:43.350786 11511 tablet_replica.cc:330] T 4187a78e6c9149fa9127b1ba98f33d2d P 7c75efa0e0da41d7883d6fc4bc7c6f89: stopping tablet replica
I20260619 18:47:43.351310 11511 raft_consensus.cc:2229] T 4187a78e6c9149fa9127b1ba98f33d2d P 7c75efa0e0da41d7883d6fc4bc7c6f89 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:47:43.352109 11511 raft_consensus.cc:2258] T 4187a78e6c9149fa9127b1ba98f33d2d P 7c75efa0e0da41d7883d6fc4bc7c6f89 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:47:43.357868 11511 ts_tablet_manager.cc:1905] T 4187a78e6c9149fa9127b1ba98f33d2d P 7c75efa0e0da41d7883d6fc4bc7c6f89: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:47:43.361617 11510 tablet_replica.cc:330] T 4187a78e6c9149fa9127b1ba98f33d2d P 69da4e79b53c43fc8becd20ad510c8b8: stopping tablet replica
I20260619 18:47:43.362099 11510 raft_consensus.cc:2229] T 4187a78e6c9149fa9127b1ba98f33d2d P 69da4e79b53c43fc8becd20ad510c8b8 [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:47:43.362964 11510 raft_consensus.cc:2258] T 4187a78e6c9149fa9127b1ba98f33d2d P 69da4e79b53c43fc8becd20ad510c8b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:47:43.367496 11510 ts_tablet_manager.cc:1905] T 4187a78e6c9149fa9127b1ba98f33d2d P 69da4e79b53c43fc8becd20ad510c8b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:47:43.371371 11509 tablet_replica.cc:330] T 4187a78e6c9149fa9127b1ba98f33d2d P 3f5657a7496c4e7b82d4244c01154242: stopping tablet replica
I20260619 18:47:43.371842 11509 raft_consensus.cc:2229] T 4187a78e6c9149fa9127b1ba98f33d2d P 3f5657a7496c4e7b82d4244c01154242 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:47:43.372614 11509 raft_consensus.cc:2258] T 4187a78e6c9149fa9127b1ba98f33d2d P 3f5657a7496c4e7b82d4244c01154242 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:47:43.374564 11509 ts_tablet_manager.cc:1905] T 4187a78e6c9149fa9127b1ba98f33d2d P 3f5657a7496c4e7b82d4244c01154242: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:47:43.375705 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 10168
I20260619 18:47:43.380686 10230 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:47:43.381223 11511 ts_tablet_manager.cc:1918] T 4187a78e6c9149fa9127b1ba98f33d2d P 7c75efa0e0da41d7883d6fc4bc7c6f89: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:47:43.381484 11511 log.cc:1196] T 4187a78e6c9149fa9127b1ba98f33d2d P 7c75efa0e0da41d7883d6fc4bc7c6f89: Deleting WAL directory at /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/ts-2/wal/wals/4187a78e6c9149fa9127b1ba98f33d2d
I20260619 18:47:43.382190 11511 ts_tablet_manager.cc:1939] T 4187a78e6c9149fa9127b1ba98f33d2d P 7c75efa0e0da41d7883d6fc4bc7c6f89: Deleting consensus metadata
I20260619 18:47:43.386294 11510 ts_tablet_manager.cc:1918] T 4187a78e6c9149fa9127b1ba98f33d2d P 69da4e79b53c43fc8becd20ad510c8b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:47:43.386544 11510 log.cc:1196] T 4187a78e6c9149fa9127b1ba98f33d2d P 69da4e79b53c43fc8becd20ad510c8b8: Deleting WAL directory at /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/ts-0/wal/wals/4187a78e6c9149fa9127b1ba98f33d2d
I20260619 18:47:43.387269 11510 ts_tablet_manager.cc:1939] T 4187a78e6c9149fa9127b1ba98f33d2d P 69da4e79b53c43fc8becd20ad510c8b8: Deleting consensus metadata
I20260619 18:47:43.399014 11509 ts_tablet_manager.cc:1918] T 4187a78e6c9149fa9127b1ba98f33d2d P 3f5657a7496c4e7b82d4244c01154242: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:47:43.399319 11509 log.cc:1196] T 4187a78e6c9149fa9127b1ba98f33d2d P 3f5657a7496c4e7b82d4244c01154242: Deleting WAL directory at /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/ts-1/wal/wals/4187a78e6c9149fa9127b1ba98f33d2d
I20260619 18:47:43.403309 11457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:43.346722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45712 (local address 127.3.243.125:37445)
0619 18:47:43.346894 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:43.346900 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:43.370381 (+ 23481us) server_negotiation.cc:407] Connection header received
0619 18:47:43.370595 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:43.370600 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:43.370697 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:43.370794 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:43.376847 (+ 6053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:43.378323 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:43.380721 (+ 2398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:43.381544 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:43.381687 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:43.393139 (+ 11452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:43.393146 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:43.393151 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:43.393193 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:43.397968 (+ 4775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:43.400259 (+ 2291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:43.400265 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:43.400269 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:43.400483 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:43.401024 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:43.401030 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:43.401033 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:43.401504 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:43.401817 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:43.402535 (+ 718us) server_negotiation.cc:299] Negotiation successful
0619 18:47:43.402797 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 18:47:43.403746 11509 ts_tablet_manager.cc:1939] T 4187a78e6c9149fa9127b1ba98f33d2d P 3f5657a7496c4e7b82d4244c01154242: Deleting consensus metadata
I20260619 18:47:43.425724 11458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:43.374594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39942 (local address 127.3.243.124:32923)
0619 18:47:43.374840 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:43.374846 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:43.405610 (+ 30764us) server_negotiation.cc:407] Connection header received
0619 18:47:43.405687 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:43.405692 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:43.405788 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:43.405895 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:43.408461 (+ 2566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:43.409846 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:43.412170 (+ 2324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:43.412975 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:43.413077 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:43.417623 (+ 4546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:43.417629 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:43.417634 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:43.417682 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:43.422263 (+ 4581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:43.423175 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:43.423180 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:43.423184 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:43.423394 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:43.423844 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:43.423849 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:43.423852 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:43.424316 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:43.424630 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:43.425012 (+ 382us) server_negotiation.cc:299] Negotiation successful
0619 18:47:43.425226 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260619 18:47:43.518368 10186 catalog_manager.cc:4841] TS 3f5657a7496c4e7b82d4244c01154242 (127.3.243.66:37147): tablet 4187a78e6c9149fa9127b1ba98f33d2d (table test-table [id=d8926a32c03245168a288f2e22c1ce01]) successfully deleted
I20260619 18:47:43.520345 10187 catalog_manager.cc:4841] TS 7c75efa0e0da41d7883d6fc4bc7c6f89 (127.3.243.67:36847): tablet 4187a78e6c9149fa9127b1ba98f33d2d (table test-table [id=d8926a32c03245168a288f2e22c1ce01]) successfully deleted
I20260619 18:47:43.520350 10185 catalog_manager.cc:4841] TS 69da4e79b53c43fc8becd20ad510c8b8 (127.3.243.65:34185): tablet 4187a78e6c9149fa9127b1ba98f33d2d (table test-table [id=d8926a32c03245168a288f2e22c1ce01]) successfully deleted
I20260619 18:47:43.521368 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 10168
I20260619 18:47:43.567683 11457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:43.550620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45716 (local address 127.3.243.125:37445)
0619 18:47:43.550855 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:43.550862 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:43.551727 (+ 865us) server_negotiation.cc:407] Connection header received
0619 18:47:43.551993 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:43.551998 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:43.552084 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:43.552192 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:43.554444 (+ 2252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:43.555620 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:43.557686 (+ 2066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:43.558383 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:43.558484 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:43.562002 (+ 3518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:43.562006 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:43.562010 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:43.562040 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:43.564932 (+ 2892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:43.565569 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:43.565574 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:43.565576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:43.565733 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:43.566126 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:43.566131 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:43.566135 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:43.566480 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:43.566725 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:43.567100 (+ 375us) server_negotiation.cc:299] Negotiation successful
0619 18:47:43.567342 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 18:47:43.575212 11458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:43.551718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39950 (local address 127.3.243.124:32923)
0619 18:47:43.552007 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:43.552013 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:43.560684 (+ 8671us) server_negotiation.cc:407] Connection header received
0619 18:47:43.560822 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:43.560828 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:43.560912 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:43.561017 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:43.562792 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:43.563847 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:43.565705 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:43.566264 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:43.566383 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:43.570220 (+ 3837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:43.570225 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:43.570228 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:43.570260 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:43.572665 (+ 2405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:43.573504 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:43.573510 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:43.573513 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:43.573642 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:43.574007 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:43.574010 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:43.574012 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:43.574263 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:43.574436 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:43.574810 (+ 374us) server_negotiation.cc:299] Negotiation successful
0619 18:47:43.574972 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 18:47:44.324581 11521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:44.323910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:41477 (local address 127.3.243.65:50605)
0619 18:47:44.324324 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:44.324432 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:41477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":199,"threads_started":1}
W20260619 18:47:44.324930 11166 heartbeater.cc:644] Failed to heartbeat to 127.3.243.126:41477 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.243.126:41477: connect: Connection refused (error 111)
I20260619 18:47:44.328306 11522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:44.327673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:41477 (local address 127.3.243.66:38927)
0619 18:47:44.328018 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:44.328120 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:41477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":166,"threads_started":1}
W20260619 18:47:44.328627 11312 heartbeater.cc:644] Failed to heartbeat to 127.3.243.126:41477 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.243.126:41477: connect: Connection refused (error 111)
I20260619 18:47:44.331007 11523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:44.330293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:41477 (local address 127.3.243.67:57827)
0619 18:47:44.330652 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:44.330796 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:41477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":173,"threads_started":1}
W20260619 18:47:44.331316 11453 heartbeater.cc:644] Failed to heartbeat to 127.3.243.126:41477 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.243.126:41477: connect: Connection refused (error 111)
I20260619 18:47:44.841266 11524 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed237dce23ef407fa74b615f8f6c914f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3e612a9d44274aadb7de143295bf9e38)
I20260619 18:47:44.841466 11525 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae1e1640d5134c40a77de2ae1fd29530 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3e612a9d44274aadb7de143295bf9e38)
I20260619 18:47:44.841549 11524 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed237dce23ef407fa74b615f8f6c914f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3e612a9d44274aadb7de143295bf9e38" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 41477 } } peers { permanent_uuid: "ae1e1640d5134c40a77de2ae1fd29530" member_type: VOTER last_known_addr { host: "127.3.243.125" port: 37445 } attrs { promote: false } } peers { permanent_uuid: "ed237dce23ef407fa74b615f8f6c914f" member_type: VOTER last_known_addr { host: "127.3.243.124" port: 32923 } attrs { promote: false } }
I20260619 18:47:44.841740 11525 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae1e1640d5134c40a77de2ae1fd29530 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3e612a9d44274aadb7de143295bf9e38" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 41477 } } peers { permanent_uuid: "ae1e1640d5134c40a77de2ae1fd29530" member_type: VOTER last_known_addr { host: "127.3.243.125" port: 37445 } attrs { promote: false } } peers { permanent_uuid: "ed237dce23ef407fa74b615f8f6c914f" member_type: VOTER last_known_addr { host: "127.3.243.124" port: 32923 } attrs { promote: false } }
I20260619 18:47:44.843065 11524 leader_election.cc:288] T 00000000000000000000000000000000 P ed237dce23ef407fa74b615f8f6c914f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3e612a9d44274aadb7de143295bf9e38 (127.3.243.126:41477), ae1e1640d5134c40a77de2ae1fd29530 (127.3.243.125:37445)
I20260619 18:47:44.844255 11527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:44.843187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:41477 (local address 127.0.0.1:33430)
0619 18:47:44.843707 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:44.843959 (+ 252us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:41477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":219,"threads_started":1}
I20260619 18:47:44.843892 11525 leader_election.cc:288] T 00000000000000000000000000000000 P ae1e1640d5134c40a77de2ae1fd29530 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3e612a9d44274aadb7de143295bf9e38 (127.3.243.126:41477), ed237dce23ef407fa74b615f8f6c914f (127.3.243.124:32923)
W20260619 18:47:44.844502 10974 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.243.126:41477: connect: Connection refused (error 111)
I20260619 18:47:44.845366 11526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:44.843179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:41477 (local address 127.0.0.1:33426)
0619 18:47:44.844893 (+ 1714us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:44.845087 (+ 194us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:41477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1591,"thread_start_us":1433,"threads_started":1}
I20260619 18:47:44.846974 11527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:44.846652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:41477 (local address 127.0.0.1:33442)
0619 18:47:44.846782 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:44.846869 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:41477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260619 18:47:44.847152 10974 leader_election.cc:334] T 00000000000000000000000000000000 P ed237dce23ef407fa74b615f8f6c914f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3e612a9d44274aadb7de143295bf9e38 (127.3.243.126:41477): Network error: Client connection negotiation failed: client connection to 127.3.243.126:41477: connect: Connection refused (error 111)
W20260619 18:47:44.847172 10783 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.243.126:41477: connect: Connection refused (error 111)
I20260619 18:47:44.850183 11530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:44.849495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:41477 (local address 127.0.0.1:33458)
0619 18:47:44.849751 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:44.849928 (+ 177us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:41477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
W20260619 18:47:44.850373 10783 leader_election.cc:334] T 00000000000000000000000000000000 P ae1e1640d5134c40a77de2ae1fd29530 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3e612a9d44274aadb7de143295bf9e38 (127.3.243.126:41477): Network error: Client connection negotiation failed: client connection to 127.3.243.126:41477: connect: Connection refused (error 111)
I20260619 18:47:44.853895 11528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:44.843251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.125:37445 (local address 127.0.0.1:45720)
0619 18:47:44.843964 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:44.843979 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:44.844110 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:44.847520 (+ 3410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:44.847525 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:44.847557 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:47:44.847869 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:44.847881 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:44.850631 (+ 2750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:44.850636 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:44.851958 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:44.851970 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:44.853083 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:44.853088 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:44.853287 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:44.853311 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 18:47:44.853391 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 18:47:44.853509 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":409,"threads_started":1}
I20260619 18:47:44.853937 11532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:44.846541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45720 (local address 127.3.243.125:37445)
0619 18:47:44.847196 (+ 655us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:44.847202 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:44.847219 (+ 17us) server_negotiation.cc:407] Connection header received
0619 18:47:44.847293 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:44.847297 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:44.847363 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:44.847462 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:47:44.848069 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:44.850463 (+ 2394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:44.852111 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:44.852923 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:44.853032 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:44.853118 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:44.853436 (+ 318us) server_negotiation.cc:299] Negotiation successful
0619 18:47:44.853564 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":378,"threads_started":1}
I20260619 18:47:44.853983 11526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:44.844985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.124:32923 (local address 127.0.0.1:39960)
0619 18:47:44.845565 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:44.845583 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:44.845708 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:44.847853 (+ 2145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:44.847858 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:44.847900 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:47:44.849211 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:44.849223 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:44.850845 (+ 1622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:44.850850 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:44.852095 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:44.852106 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:44.853215 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:44.853221 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:44.853380 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:44.853402 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 18:47:44.853490 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 18:47:44.853627 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"mutex_wait_us":1,"thread_start_us":1351,"threads_started":1}
I20260619 18:47:44.854200 11531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:44.845241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39960 (local address 127.3.243.124:32923)
0619 18:47:44.847550 (+ 2309us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:44.847555 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:44.847574 (+ 19us) server_negotiation.cc:407] Connection header received
0619 18:47:44.847615 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:44.847619 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:44.847681 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:44.848035 (+ 354us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:47:44.849387 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:44.850726 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:44.852254 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:44.853046 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:44.853151 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:44.853254 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:44.853532 (+ 278us) server_negotiation.cc:299] Negotiation successful
0619 18:47:44.853656 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2184,"thread_start_us":1399,"threads_started":1}
I20260619 18:47:44.854718 10807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ed237dce23ef407fa74b615f8f6c914f" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ae1e1640d5134c40a77de2ae1fd29530" is_pre_election: true
I20260619 18:47:44.854732 10998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ae1e1640d5134c40a77de2ae1fd29530" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ed237dce23ef407fa74b615f8f6c914f" is_pre_election: true
I20260619 18:47:44.854982 10998 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ed237dce23ef407fa74b615f8f6c914f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae1e1640d5134c40a77de2ae1fd29530 in term 1.
I20260619 18:47:44.854992 10807 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ae1e1640d5134c40a77de2ae1fd29530 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed237dce23ef407fa74b615f8f6c914f in term 1.
I20260619 18:47:44.855423 10783 leader_election.cc:302] T 00000000000000000000000000000000 P ae1e1640d5134c40a77de2ae1fd29530 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae1e1640d5134c40a77de2ae1fd29530, ed237dce23ef407fa74b615f8f6c914f; no voters: 3e612a9d44274aadb7de143295bf9e38
I20260619 18:47:44.855423 10975 leader_election.cc:302] T 00000000000000000000000000000000 P ed237dce23ef407fa74b615f8f6c914f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae1e1640d5134c40a77de2ae1fd29530, ed237dce23ef407fa74b615f8f6c914f; no voters: 3e612a9d44274aadb7de143295bf9e38
I20260619 18:47:44.855757 11525 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae1e1640d5134c40a77de2ae1fd29530 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260619 18:47:44.855787 11524 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed237dce23ef407fa74b615f8f6c914f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260619 18:47:44.855888 11524 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed237dce23ef407fa74b615f8f6c914f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3e612a9d44274aadb7de143295bf9e38)
I20260619 18:47:44.855885 11525 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae1e1640d5134c40a77de2ae1fd29530 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3e612a9d44274aadb7de143295bf9e38)
I20260619 18:47:44.855988 11525 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae1e1640d5134c40a77de2ae1fd29530 [term 1 FOLLOWER]: Advancing to term 2
I20260619 18:47:44.855969 11524 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed237dce23ef407fa74b615f8f6c914f [term 1 FOLLOWER]: Advancing to term 2
I20260619 18:47:44.858919 11524 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed237dce23ef407fa74b615f8f6c914f [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3e612a9d44274aadb7de143295bf9e38" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 41477 } } peers { permanent_uuid: "ae1e1640d5134c40a77de2ae1fd29530" member_type: VOTER last_known_addr { host: "127.3.243.125" port: 37445 } attrs { promote: false } } peers { permanent_uuid: "ed237dce23ef407fa74b615f8f6c914f" member_type: VOTER last_known_addr { host: "127.3.243.124" port: 32923 } attrs { promote: false } }
I20260619 18:47:44.858922 11525 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae1e1640d5134c40a77de2ae1fd29530 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3e612a9d44274aadb7de143295bf9e38" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 41477 } } peers { permanent_uuid: "ae1e1640d5134c40a77de2ae1fd29530" member_type: VOTER last_known_addr { host: "127.3.243.125" port: 37445 } attrs { promote: false } } peers { permanent_uuid: "ed237dce23ef407fa74b615f8f6c914f" member_type: VOTER last_known_addr { host: "127.3.243.124" port: 32923 } attrs { promote: false } }
I20260619 18:47:44.859570 11525 leader_election.cc:288] T 00000000000000000000000000000000 P ae1e1640d5134c40a77de2ae1fd29530 [CANDIDATE]: Term 2 election: Requested vote from peers 3e612a9d44274aadb7de143295bf9e38 (127.3.243.126:41477), ed237dce23ef407fa74b615f8f6c914f (127.3.243.124:32923)
I20260619 18:47:44.859584 11524 leader_election.cc:288] T 00000000000000000000000000000000 P ed237dce23ef407fa74b615f8f6c914f [CANDIDATE]: Term 2 election: Requested vote from peers 3e612a9d44274aadb7de143295bf9e38 (127.3.243.126:41477), ae1e1640d5134c40a77de2ae1fd29530 (127.3.243.125:37445)
I20260619 18:47:44.860038 11526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:44.859682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:41477 (local address 127.0.0.1:33472)
0619 18:47:44.859821 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:44.859901 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:41477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260619 18:47:44.860046 11528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:44.859692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:41477 (local address 127.0.0.1:33474)
0619 18:47:44.859821 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:44.859902 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:41477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260619 18:47:44.860196 10998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ae1e1640d5134c40a77de2ae1fd29530" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ed237dce23ef407fa74b615f8f6c914f"
I20260619 18:47:44.860392 10807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ed237dce23ef407fa74b615f8f6c914f" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ae1e1640d5134c40a77de2ae1fd29530"
I20260619 18:47:44.860466 10998 raft_consensus.cc:2379] T 00000000000000000000000000000000 P ed237dce23ef407fa74b615f8f6c914f [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate ae1e1640d5134c40a77de2ae1fd29530 in current term 2: Already voted for candidate ed237dce23ef407fa74b615f8f6c914f in this term.
I20260619 18:47:44.860695 10807 raft_consensus.cc:2379] T 00000000000000000000000000000000 P ae1e1640d5134c40a77de2ae1fd29530 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate ed237dce23ef407fa74b615f8f6c914f in current term 2: Already voted for candidate ae1e1640d5134c40a77de2ae1fd29530 in this term.
I20260619 18:47:44.861267 11528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:44.860962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:41477 (local address 127.0.0.1:33496)
0619 18:47:44.861094 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:44.861163 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:41477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260619 18:47:44.861407 11526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:44.860980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:41477 (local address 127.0.0.1:33512)
0619 18:47:44.861143 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:44.861218 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:41477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260619 18:47:44.861443 10974 leader_election.cc:334] T 00000000000000000000000000000000 P ed237dce23ef407fa74b615f8f6c914f [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 3e612a9d44274aadb7de143295bf9e38 (127.3.243.126:41477): Network error: Client connection negotiation failed: client connection to 127.3.243.126:41477: connect: Connection refused (error 111)
I20260619 18:47:44.861591 10975 leader_election.cc:302] T 00000000000000000000000000000000 P ed237dce23ef407fa74b615f8f6c914f [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ed237dce23ef407fa74b615f8f6c914f; no voters: 3e612a9d44274aadb7de143295bf9e38, ae1e1640d5134c40a77de2ae1fd29530
I20260619 18:47:44.861816 11524 raft_consensus.cc:2726] T 00000000000000000000000000000000 P ed237dce23ef407fa74b615f8f6c914f [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260619 18:47:44.861976 10783 leader_election.cc:334] T 00000000000000000000000000000000 P ae1e1640d5134c40a77de2ae1fd29530 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 3e612a9d44274aadb7de143295bf9e38 (127.3.243.126:41477): Network error: Client connection negotiation failed: client connection to 127.3.243.126:41477: connect: Connection refused (error 111)
I20260619 18:47:44.862095 10783 leader_election.cc:302] T 00000000000000000000000000000000 P ae1e1640d5134c40a77de2ae1fd29530 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae1e1640d5134c40a77de2ae1fd29530; no voters: 3e612a9d44274aadb7de143295bf9e38, ed237dce23ef407fa74b615f8f6c914f
I20260619 18:47:44.862344 11525 raft_consensus.cc:2726] T 00000000000000000000000000000000 P ae1e1640d5134c40a77de2ae1fd29530 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260619 18:47:45.326195 11534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:45.325499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:41477 (local address 127.3.243.65:49949)
0619 18:47:45.325902 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:45.325993 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:41477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":206,"threads_started":1}
I20260619 18:47:45.329777 11535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:45.329139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:41477 (local address 127.3.243.66:55583)
0619 18:47:45.329551 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:45.329628 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:41477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":217,"threads_started":1}
I20260619 18:47:45.332628 11536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:45.331922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:41477 (local address 127.3.243.67:43563)
0619 18:47:45.332352 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:45.332446 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:41477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":233,"threads_started":1}
I20260619 18:47:46.327533 11537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:46.326898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:41477 (local address 127.3.243.65:50079)
0619 18:47:46.327288 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:46.327372 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:41477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":185,"threads_started":1}
I20260619 18:47:46.331034 11538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:46.330427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:41477 (local address 127.3.243.66:52435)
0619 18:47:46.330789 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:46.330870 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:41477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":196,"threads_started":1}
I20260619 18:47:46.334151 11539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:46.333402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:41477 (local address 127.3.243.67:54513)
0619 18:47:46.333861 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:46.333948 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:41477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":246,"threads_started":1}
I20260619 18:47:46.815428 11540 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed237dce23ef407fa74b615f8f6c914f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:47:46.815665 11540 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed237dce23ef407fa74b615f8f6c914f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3e612a9d44274aadb7de143295bf9e38" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 41477 } } peers { permanent_uuid: "ae1e1640d5134c40a77de2ae1fd29530" member_type: VOTER last_known_addr { host: "127.3.243.125" port: 37445 } attrs { promote: false } } peers { permanent_uuid: "ed237dce23ef407fa74b615f8f6c914f" member_type: VOTER last_known_addr { host: "127.3.243.124" port: 32923 } attrs { promote: false } }
I20260619 18:47:46.816316 11540 leader_election.cc:288] T 00000000000000000000000000000000 P ed237dce23ef407fa74b615f8f6c914f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3e612a9d44274aadb7de143295bf9e38 (127.3.243.126:41477), ae1e1640d5134c40a77de2ae1fd29530 (127.3.243.125:37445)
I20260619 18:47:46.816867 10807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ed237dce23ef407fa74b615f8f6c914f" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ae1e1640d5134c40a77de2ae1fd29530" is_pre_election: true
I20260619 18:47:46.817104 10807 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ae1e1640d5134c40a77de2ae1fd29530 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed237dce23ef407fa74b615f8f6c914f in term 2.
I20260619 18:47:46.817267 11541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:46.816515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:41477 (local address 127.0.0.1:33780)
0619 18:47:46.816971 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:46.817065 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:41477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":237,"threads_started":1}
I20260619 18:47:46.817571 10975 leader_election.cc:302] T 00000000000000000000000000000000 P ed237dce23ef407fa74b615f8f6c914f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae1e1640d5134c40a77de2ae1fd29530, ed237dce23ef407fa74b615f8f6c914f; no voters:
I20260619 18:47:46.817834 11540 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed237dce23ef407fa74b615f8f6c914f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260619 18:47:46.817993 11540 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed237dce23ef407fa74b615f8f6c914f [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:47:46.818095 11540 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed237dce23ef407fa74b615f8f6c914f [term 2 FOLLOWER]: Advancing to term 3
I20260619 18:47:46.818876 11541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:46.818509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:41477 (local address 127.0.0.1:33794)
0619 18:47:46.818670 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:46.818747 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:41477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260619 18:47:46.819033 10974 leader_election.cc:334] T 00000000000000000000000000000000 P ed237dce23ef407fa74b615f8f6c914f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3e612a9d44274aadb7de143295bf9e38 (127.3.243.126:41477): Network error: Client connection negotiation failed: client connection to 127.3.243.126:41477: connect: Connection refused (error 111)
I20260619 18:47:46.820472 11540 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed237dce23ef407fa74b615f8f6c914f [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3e612a9d44274aadb7de143295bf9e38" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 41477 } } peers { permanent_uuid: "ae1e1640d5134c40a77de2ae1fd29530" member_type: VOTER last_known_addr { host: "127.3.243.125" port: 37445 } attrs { promote: false } } peers { permanent_uuid: "ed237dce23ef407fa74b615f8f6c914f" member_type: VOTER last_known_addr { host: "127.3.243.124" port: 32923 } attrs { promote: false } }
I20260619 18:47:46.821005 11540 leader_election.cc:288] T 00000000000000000000000000000000 P ed237dce23ef407fa74b615f8f6c914f [CANDIDATE]: Term 3 election: Requested vote from peers 3e612a9d44274aadb7de143295bf9e38 (127.3.243.126:41477), ae1e1640d5134c40a77de2ae1fd29530 (127.3.243.125:37445)
I20260619 18:47:46.821446 11541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:46.821104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:41477 (local address 127.0.0.1:33798)
0619 18:47:46.821241 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:46.821322 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:41477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260619 18:47:46.821465 10807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ed237dce23ef407fa74b615f8f6c914f" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ae1e1640d5134c40a77de2ae1fd29530"
I20260619 18:47:46.821658 10807 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae1e1640d5134c40a77de2ae1fd29530 [term 2 FOLLOWER]: Advancing to term 3
I20260619 18:47:46.822322 11541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:46.822037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:41477 (local address 127.0.0.1:33802)
0619 18:47:46.822158 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:46.822221 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:41477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260619 18:47:46.822475 10974 leader_election.cc:334] T 00000000000000000000000000000000 P ed237dce23ef407fa74b615f8f6c914f [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 3e612a9d44274aadb7de143295bf9e38 (127.3.243.126:41477): Network error: Client connection negotiation failed: client connection to 127.3.243.126:41477: connect: Connection refused (error 111)
I20260619 18:47:46.823830 10807 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ae1e1640d5134c40a77de2ae1fd29530 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed237dce23ef407fa74b615f8f6c914f in term 3.
I20260619 18:47:46.824185 10975 leader_election.cc:302] T 00000000000000000000000000000000 P ed237dce23ef407fa74b615f8f6c914f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae1e1640d5134c40a77de2ae1fd29530, ed237dce23ef407fa74b615f8f6c914f; no voters: 3e612a9d44274aadb7de143295bf9e38
I20260619 18:47:46.824456 11540 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed237dce23ef407fa74b615f8f6c914f [term 3 FOLLOWER]: Leader election won for term 3
I20260619 18:47:46.824841 11540 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed237dce23ef407fa74b615f8f6c914f [term 3 LEADER]: Becoming Leader. State: Replica: ed237dce23ef407fa74b615f8f6c914f, State: Running, Role: LEADER
I20260619 18:47:46.825254 11540 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed237dce23ef407fa74b615f8f6c914f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3e612a9d44274aadb7de143295bf9e38" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 41477 } } peers { permanent_uuid: "ae1e1640d5134c40a77de2ae1fd29530" member_type: VOTER last_known_addr { host: "127.3.243.125" port: 37445 } attrs { promote: false } } peers { permanent_uuid: "ed237dce23ef407fa74b615f8f6c914f" member_type: VOTER last_known_addr { host: "127.3.243.124" port: 32923 } attrs { promote: false } }
I20260619 18:47:46.828317 11543 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed237dce23ef407fa74b615f8f6c914f [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed237dce23ef407fa74b615f8f6c914f. Latest consensus state: current_term: 3 leader_uuid: "ed237dce23ef407fa74b615f8f6c914f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3e612a9d44274aadb7de143295bf9e38" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 41477 } } peers { permanent_uuid: "ae1e1640d5134c40a77de2ae1fd29530" member_type: VOTER last_known_addr { host: "127.3.243.125" port: 37445 } attrs { promote: false } } peers { permanent_uuid: "ed237dce23ef407fa74b615f8f6c914f" member_type: VOTER last_known_addr { host: "127.3.243.124" port: 32923 } attrs { promote: false } } }
I20260619 18:47:46.828550 11543 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed237dce23ef407fa74b615f8f6c914f [sys.catalog]: This master's current role is: LEADER
I20260619 18:47:46.829319 11545 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:47:46.832614 11545 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:47:46.833375 11545 catalog_manager.cc:1259] Loaded cluster ID: 301fc5ce1f33458494634dbfacabe160
I20260619 18:47:46.833442 11545 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:47:46.834592 11545 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:47:46.835356 11545 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ed237dce23ef407fa74b615f8f6c914f: Loaded TSK: 0
I20260619 18:47:46.836176 11545 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:47:46.915028 10987 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39950:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:47:46.921741 10987 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:47:46.925849 11541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:46.925410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:41477 (local address 127.0.0.1:33826)
0619 18:47:46.925604 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:46.925713 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:41477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260619 18:47:46.925962 10807 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ae1e1640d5134c40a77de2ae1fd29530 [term 3 FOLLOWER]: Refusing update from remote peer ed237dce23ef407fa74b615f8f6c914f: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260619 18:47:46.926788 11541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:46.926425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:41477 (local address 127.0.0.1:33828)
0619 18:47:46.926587 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:46.926668 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:41477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260619 18:47:46.926659 11543 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ed237dce23ef407fa74b615f8f6c914f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae1e1640d5134c40a77de2ae1fd29530" member_type: VOTER last_known_addr { host: "127.3.243.125" port: 37445 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260619 18:47:46.926947 10974 consensus_peers.cc:487] T 00000000000000000000000000000000 P ed237dce23ef407fa74b615f8f6c914f -> Peer 3e612a9d44274aadb7de143295bf9e38 (127.3.243.126:41477): Couldn't send request to peer 3e612a9d44274aadb7de143295bf9e38. Status: Network error: Client connection negotiation failed: client connection to 127.3.243.126:41477: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260619 18:47:46.929778 11547 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae1e1640d5134c40a77de2ae1fd29530 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed237dce23ef407fa74b615f8f6c914f. Latest consensus state: current_term: 3 leader_uuid: "ed237dce23ef407fa74b615f8f6c914f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3e612a9d44274aadb7de143295bf9e38" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 41477 } } peers { permanent_uuid: "ae1e1640d5134c40a77de2ae1fd29530" member_type: VOTER last_known_addr { host: "127.3.243.125" port: 37445 } attrs { promote: false } } peers { permanent_uuid: "ed237dce23ef407fa74b615f8f6c914f" member_type: VOTER last_known_addr { host: "127.3.243.124" port: 32923 } attrs { promote: false } } }
I20260619 18:47:46.930033 11547 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae1e1640d5134c40a77de2ae1fd29530 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:47:46.930752 11540 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed237dce23ef407fa74b615f8f6c914f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "ed237dce23ef407fa74b615f8f6c914f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3e612a9d44274aadb7de143295bf9e38" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 41477 } } peers { permanent_uuid: "ae1e1640d5134c40a77de2ae1fd29530" member_type: VOTER last_known_addr { host: "127.3.243.125" port: 37445 } attrs { promote: false } } peers { permanent_uuid: "ed237dce23ef407fa74b615f8f6c914f" member_type: VOTER last_known_addr { host: "127.3.243.124" port: 32923 } attrs { promote: false } } }
I20260619 18:47:46.931082 11540 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed237dce23ef407fa74b615f8f6c914f [sys.catalog]: This master's current role is: LEADER
I20260619 18:47:46.931308 11548 mvcc.cc:205] Tried to move back new op lower bound from 7298641374921732096 to 7298641374521794560. Current Snapshot: MvccSnapshot[applied={T|T < 7298641374921732096}]
I20260619 18:47:46.932257 11547 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae1e1640d5134c40a77de2ae1fd29530 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "ed237dce23ef407fa74b615f8f6c914f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3e612a9d44274aadb7de143295bf9e38" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 41477 } } peers { permanent_uuid: "ae1e1640d5134c40a77de2ae1fd29530" member_type: VOTER last_known_addr { host: "127.3.243.125" port: 37445 } attrs { promote: false } } peers { permanent_uuid: "ed237dce23ef407fa74b615f8f6c914f" member_type: VOTER last_known_addr { host: "127.3.243.124" port: 32923 } attrs { promote: false } } }
I20260619 18:47:46.932520 11547 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae1e1640d5134c40a77de2ae1fd29530 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:47:46.950659 11555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:46.938931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50340 (local address 127.3.243.66:37147)
0619 18:47:46.939642 (+ 711us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:46.939648 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:46.939668 (+ 20us) server_negotiation.cc:407] Connection header received
0619 18:47:46.939741 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:46.939747 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:46.939823 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:46.945201 (+ 5378us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:47:46.945483 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:46.946616 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:46.948453 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:46.949757 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:46.949879 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:46.949975 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:46.950242 (+ 267us) server_negotiation.cc:299] Negotiation successful
0619 18:47:46.950376 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":485,"threads_started":1}
I20260619 18:47:46.950726 11554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:46.938614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.66:37147 (local address 127.0.0.1:50340)
0619 18:47:46.939107 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:46.939123 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:46.939234 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:46.944579 (+ 5345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:46.944585 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:46.944614 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:47:46.944986 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:46.944997 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:46.946744 (+ 1747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:46.946749 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:46.948249 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:46.948263 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:46.949930 (+ 1667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:46.949936 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:46.950094 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:46.950118 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 18:47:46.950199 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 18:47:46.950328 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378,"mutex_wait_us":19,"thread_start_us":448,"threads_started":1}
I20260619 18:47:46.951596 11247 tablet_service.cc:1467] Processing CreateTablet for tablet b7b72c7308554519b2f323cae19b41c0 (DEFAULT_TABLE table=test-table [id=3aa3d3cb858a42b3a09cc9edd65962a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:47:46.951982 11247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7b72c7308554519b2f323cae19b41c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:47:46.953433 11556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:46.940137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.65:34185 (local address 127.0.0.1:50202)
0619 18:47:46.945783 (+ 5646us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:46.945805 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:46.945914 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:46.946545 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:46.946549 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:46.946570 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:47:46.946965 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:46.946979 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:46.948497 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:46.948503 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:46.950745 (+ 2242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:46.950760 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:46.952605 (+ 1845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:46.952611 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:46.952799 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:46.952836 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 18:47:46.952922 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 18:47:46.953054 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5500,"mutex_wait_us":19,"thread_start_us":4169,"threads_started":1}
I20260619 18:47:46.954113 11557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:46.940301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50202 (local address 127.3.243.65:34185)
0619 18:47:46.946305 (+ 6004us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:46.946310 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:46.946324 (+ 14us) server_negotiation.cc:407] Connection header received
0619 18:47:46.946378 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:46.946381 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:46.946433 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:46.946513 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:47:46.947166 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:46.948321 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:46.950944 (+ 2623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:46.952225 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:46.952345 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:46.952455 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:46.953670 (+ 1215us) server_negotiation.cc:299] Negotiation successful
0619 18:47:46.953813 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5907,"thread_start_us":5803,"threads_started":1}
I20260619 18:47:46.954829 11101 tablet_service.cc:1467] Processing CreateTablet for tablet b7b72c7308554519b2f323cae19b41c0 (DEFAULT_TABLE table=test-table [id=3aa3d3cb858a42b3a09cc9edd65962a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:47:46.955202 11101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7b72c7308554519b2f323cae19b41c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:47:46.958582 11558 tablet_bootstrap.cc:492] T b7b72c7308554519b2f323cae19b41c0 P 3f5657a7496c4e7b82d4244c01154242: Bootstrap starting.
I20260619 18:47:46.959668 11541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:46.938222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.67:36847 (local address 127.0.0.1:38776)
0619 18:47:46.938412 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:46.938429 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:46.938546 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:46.953779 (+ 15233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:46.953784 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:46.953808 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:47:46.954053 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:46.954063 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:46.955464 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:46.955469 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:46.957408 (+ 1939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:46.957423 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:46.959015 (+ 1592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:46.959021 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:46.959189 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:46.959222 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 18:47:46.959284 (+ 62us) client_negotiation.cc:240] Negotiation successful
0619 18:47:46.959402 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103}
I20260619 18:47:46.960084 11558 tablet_bootstrap.cc:654] T b7b72c7308554519b2f323cae19b41c0 P 3f5657a7496c4e7b82d4244c01154242: Neither blocks nor log segments found. Creating new log.
I20260619 18:47:46.960593 11553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:46.938354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38776 (local address 127.3.243.67:36847)
0619 18:47:46.953498 (+ 15144us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:46.953505 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:46.953524 (+ 19us) server_negotiation.cc:407] Connection header received
0619 18:47:46.953585 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:46.953590 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:46.953664 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:46.953759 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:47:46.954243 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:46.955338 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:46.957590 (+ 2252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:46.958774 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:46.959971 (+ 1197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:46.960075 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:46.960135 (+ 60us) server_negotiation.cc:299] Negotiation successful
0619 18:47:46.960283 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15030,"thread_start_us":14945,"threads_started":1}
I20260619 18:47:46.961298 11388 tablet_service.cc:1467] Processing CreateTablet for tablet b7b72c7308554519b2f323cae19b41c0 (DEFAULT_TABLE table=test-table [id=3aa3d3cb858a42b3a09cc9edd65962a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:47:46.961665 11560 tablet_bootstrap.cc:492] T b7b72c7308554519b2f323cae19b41c0 P 69da4e79b53c43fc8becd20ad510c8b8: Bootstrap starting.
I20260619 18:47:46.961679 11388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7b72c7308554519b2f323cae19b41c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:47:46.962911 11560 tablet_bootstrap.cc:654] T b7b72c7308554519b2f323cae19b41c0 P 69da4e79b53c43fc8becd20ad510c8b8: Neither blocks nor log segments found. Creating new log.
I20260619 18:47:46.964890 11560 tablet_bootstrap.cc:492] T b7b72c7308554519b2f323cae19b41c0 P 69da4e79b53c43fc8becd20ad510c8b8: No bootstrap required, opened a new log
I20260619 18:47:46.965001 11560 ts_tablet_manager.cc:1397] T b7b72c7308554519b2f323cae19b41c0 P 69da4e79b53c43fc8becd20ad510c8b8: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20260619 18:47:46.965608 11560 raft_consensus.cc:348] T b7b72c7308554519b2f323cae19b41c0 P 69da4e79b53c43fc8becd20ad510c8b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c75efa0e0da41d7883d6fc4bc7c6f89" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 36847 } } peers { permanent_uuid: "3f5657a7496c4e7b82d4244c01154242" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 37147 } } peers { permanent_uuid: "69da4e79b53c43fc8becd20ad510c8b8" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 34185 } }
I20260619 18:47:46.965781 11560 raft_consensus.cc:374] T b7b72c7308554519b2f323cae19b41c0 P 69da4e79b53c43fc8becd20ad510c8b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:47:46.965858 11560 raft_consensus.cc:729] T b7b72c7308554519b2f323cae19b41c0 P 69da4e79b53c43fc8becd20ad510c8b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69da4e79b53c43fc8becd20ad510c8b8, State: Initialized, Role: FOLLOWER
I20260619 18:47:46.966075 11560 consensus_queue.cc:260] T b7b72c7308554519b2f323cae19b41c0 P 69da4e79b53c43fc8becd20ad510c8b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c75efa0e0da41d7883d6fc4bc7c6f89" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 36847 } } peers { permanent_uuid: "3f5657a7496c4e7b82d4244c01154242" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 37147 } } peers { permanent_uuid: "69da4e79b53c43fc8becd20ad510c8b8" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 34185 } }
I20260619 18:47:46.966872 11562 tablet_bootstrap.cc:492] T b7b72c7308554519b2f323cae19b41c0 P 7c75efa0e0da41d7883d6fc4bc7c6f89: Bootstrap starting.
I20260619 18:47:46.967047 11558 tablet_bootstrap.cc:492] T b7b72c7308554519b2f323cae19b41c0 P 3f5657a7496c4e7b82d4244c01154242: No bootstrap required, opened a new log
I20260619 18:47:46.967206 11558 ts_tablet_manager.cc:1397] T b7b72c7308554519b2f323cae19b41c0 P 3f5657a7496c4e7b82d4244c01154242: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20260619 18:47:46.967967 11558 raft_consensus.cc:348] T b7b72c7308554519b2f323cae19b41c0 P 3f5657a7496c4e7b82d4244c01154242 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c75efa0e0da41d7883d6fc4bc7c6f89" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 36847 } } peers { permanent_uuid: "3f5657a7496c4e7b82d4244c01154242" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 37147 } } peers { permanent_uuid: "69da4e79b53c43fc8becd20ad510c8b8" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 34185 } }
I20260619 18:47:46.968286 11558 raft_consensus.cc:374] T b7b72c7308554519b2f323cae19b41c0 P 3f5657a7496c4e7b82d4244c01154242 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:47:46.968422 11558 raft_consensus.cc:729] T b7b72c7308554519b2f323cae19b41c0 P 3f5657a7496c4e7b82d4244c01154242 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f5657a7496c4e7b82d4244c01154242, State: Initialized, Role: FOLLOWER
I20260619 18:47:46.968631 11562 tablet_bootstrap.cc:654] T b7b72c7308554519b2f323cae19b41c0 P 7c75efa0e0da41d7883d6fc4bc7c6f89: Neither blocks nor log segments found. Creating new log.
I20260619 18:47:46.968755 11558 consensus_queue.cc:260] T b7b72c7308554519b2f323cae19b41c0 P 3f5657a7496c4e7b82d4244c01154242 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c75efa0e0da41d7883d6fc4bc7c6f89" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 36847 } } peers { permanent_uuid: "3f5657a7496c4e7b82d4244c01154242" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 37147 } } peers { permanent_uuid: "69da4e79b53c43fc8becd20ad510c8b8" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 34185 } }
I20260619 18:47:46.969529 11560 ts_tablet_manager.cc:1428] T b7b72c7308554519b2f323cae19b41c0 P 69da4e79b53c43fc8becd20ad510c8b8: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260619 18:47:46.969832 11564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:46.967059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:41477 (local address 127.3.243.65:35891)
0619 18:47:46.969356 (+ 2297us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:46.969523 (+ 167us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:41477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2181,"thread_start_us":2058,"threads_started":1}
I20260619 18:47:46.972196 11167 heartbeater.cc:503] Master 127.3.243.124:32923 was elected leader, sending a full tablet report...
I20260619 18:47:46.972468 11567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:46.970426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:41477 (local address 127.3.243.65:35747)
0619 18:47:46.972016 (+ 1590us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:46.972205 (+ 189us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:41477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1484,"thread_start_us":1368,"threads_started":1}
I20260619 18:47:46.973953 11562 tablet_bootstrap.cc:492] T b7b72c7308554519b2f323cae19b41c0 P 7c75efa0e0da41d7883d6fc4bc7c6f89: No bootstrap required, opened a new log
I20260619 18:47:46.974141 11562 ts_tablet_manager.cc:1397] T b7b72c7308554519b2f323cae19b41c0 P 7c75efa0e0da41d7883d6fc4bc7c6f89: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.006s
I20260619 18:47:46.974925 11562 raft_consensus.cc:348] T b7b72c7308554519b2f323cae19b41c0 P 7c75efa0e0da41d7883d6fc4bc7c6f89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c75efa0e0da41d7883d6fc4bc7c6f89" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 36847 } } peers { permanent_uuid: "3f5657a7496c4e7b82d4244c01154242" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 37147 } } peers { permanent_uuid: "69da4e79b53c43fc8becd20ad510c8b8" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 34185 } }
I20260619 18:47:46.975152 11562 raft_consensus.cc:374] T b7b72c7308554519b2f323cae19b41c0 P 7c75efa0e0da41d7883d6fc4bc7c6f89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:47:46.975246 11562 raft_consensus.cc:729] T b7b72c7308554519b2f323cae19b41c0 P 7c75efa0e0da41d7883d6fc4bc7c6f89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c75efa0e0da41d7883d6fc4bc7c6f89, State: Initialized, Role: FOLLOWER
I20260619 18:47:46.975468 11562 consensus_queue.cc:260] T b7b72c7308554519b2f323cae19b41c0 P 7c75efa0e0da41d7883d6fc4bc7c6f89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c75efa0e0da41d7883d6fc4bc7c6f89" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 36847 } } peers { permanent_uuid: "3f5657a7496c4e7b82d4244c01154242" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 37147 } } peers { permanent_uuid: "69da4e79b53c43fc8becd20ad510c8b8" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 34185 } }
I20260619 18:47:46.975641 11558 ts_tablet_manager.cc:1428] T b7b72c7308554519b2f323cae19b41c0 P 3f5657a7496c4e7b82d4244c01154242: Time spent starting tablet: real 0.008s user 0.004s sys 0.000s
I20260619 18:47:46.977098 11562 ts_tablet_manager.cc:1428] T b7b72c7308554519b2f323cae19b41c0 P 7c75efa0e0da41d7883d6fc4bc7c6f89: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260619 18:47:46.977944 11570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:46.976528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:41477 (local address 127.3.243.67:33367)
0619 18:47:46.977571 (+ 1043us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:46.977704 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:41477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":912,"thread_start_us":811,"threads_started":1}
I20260619 18:47:46.980674 11454 heartbeater.cc:503] Master 127.3.243.124:32923 was elected leader, sending a full tablet report...
I20260619 18:47:46.980948 11571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:46.978430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:41477 (local address 127.3.243.67:49133)
0619 18:47:46.980529 (+ 2099us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:46.980692 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:41477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1994,"thread_start_us":1880,"threads_started":1}
I20260619 18:47:46.984886 11569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:46.975725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:41477 (local address 127.3.243.66:47561)
0619 18:47:46.984524 (+ 8799us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:46.984638 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:41477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":8691,"thread_start_us":8566,"threads_started":1}
I20260619 18:47:46.985760 11569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:46.985456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:41477 (local address 127.3.243.66:39657)
0619 18:47:46.985588 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:46.985655 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:41477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260619 18:47:46.985781 11313 heartbeater.cc:503] Master 127.3.243.124:32923 was elected leader, sending a full tablet report...
I20260619 18:47:47.106935 11563 raft_consensus.cc:482] T b7b72c7308554519b2f323cae19b41c0 P 69da4e79b53c43fc8becd20ad510c8b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:47:47.107151 11563 raft_consensus.cc:504] T b7b72c7308554519b2f323cae19b41c0 P 69da4e79b53c43fc8becd20ad510c8b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c75efa0e0da41d7883d6fc4bc7c6f89" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 36847 } } peers { permanent_uuid: "3f5657a7496c4e7b82d4244c01154242" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 37147 } } peers { permanent_uuid: "69da4e79b53c43fc8becd20ad510c8b8" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 34185 } }
I20260619 18:47:47.107928 11563 leader_election.cc:288] T b7b72c7308554519b2f323cae19b41c0 P 69da4e79b53c43fc8becd20ad510c8b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7c75efa0e0da41d7883d6fc4bc7c6f89 (127.3.243.67:36847), 3f5657a7496c4e7b82d4244c01154242 (127.3.243.66:37147)
I20260619 18:47:47.108500 11408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7b72c7308554519b2f323cae19b41c0" candidate_uuid: "69da4e79b53c43fc8becd20ad510c8b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c75efa0e0da41d7883d6fc4bc7c6f89" is_pre_election: true
I20260619 18:47:47.108547 11267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7b72c7308554519b2f323cae19b41c0" candidate_uuid: "69da4e79b53c43fc8becd20ad510c8b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f5657a7496c4e7b82d4244c01154242" is_pre_election: true
I20260619 18:47:47.108744 11408 raft_consensus.cc:2454] T b7b72c7308554519b2f323cae19b41c0 P 7c75efa0e0da41d7883d6fc4bc7c6f89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69da4e79b53c43fc8becd20ad510c8b8 in term 0.
I20260619 18:47:47.108801 11267 raft_consensus.cc:2454] T b7b72c7308554519b2f323cae19b41c0 P 3f5657a7496c4e7b82d4244c01154242 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69da4e79b53c43fc8becd20ad510c8b8 in term 0.
I20260619 18:47:47.109189 11057 leader_election.cc:302] T b7b72c7308554519b2f323cae19b41c0 P 69da4e79b53c43fc8becd20ad510c8b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69da4e79b53c43fc8becd20ad510c8b8, 7c75efa0e0da41d7883d6fc4bc7c6f89; no voters:
I20260619 18:47:47.109447 11563 raft_consensus.cc:2781] T b7b72c7308554519b2f323cae19b41c0 P 69da4e79b53c43fc8becd20ad510c8b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:47:47.109566 11563 raft_consensus.cc:482] T b7b72c7308554519b2f323cae19b41c0 P 69da4e79b53c43fc8becd20ad510c8b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:47:47.109663 11563 raft_consensus.cc:3037] T b7b72c7308554519b2f323cae19b41c0 P 69da4e79b53c43fc8becd20ad510c8b8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:47:47.111644 11563 raft_consensus.cc:504] T b7b72c7308554519b2f323cae19b41c0 P 69da4e79b53c43fc8becd20ad510c8b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c75efa0e0da41d7883d6fc4bc7c6f89" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 36847 } } peers { permanent_uuid: "3f5657a7496c4e7b82d4244c01154242" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 37147 } } peers { permanent_uuid: "69da4e79b53c43fc8becd20ad510c8b8" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 34185 } }
I20260619 18:47:47.112200 11563 leader_election.cc:288] T b7b72c7308554519b2f323cae19b41c0 P 69da4e79b53c43fc8becd20ad510c8b8 [CANDIDATE]: Term 1 election: Requested vote from peers 7c75efa0e0da41d7883d6fc4bc7c6f89 (127.3.243.67:36847), 3f5657a7496c4e7b82d4244c01154242 (127.3.243.66:37147)
I20260619 18:47:47.112571 11408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7b72c7308554519b2f323cae19b41c0" candidate_uuid: "69da4e79b53c43fc8becd20ad510c8b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c75efa0e0da41d7883d6fc4bc7c6f89"
I20260619 18:47:47.112661 11267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7b72c7308554519b2f323cae19b41c0" candidate_uuid: "69da4e79b53c43fc8becd20ad510c8b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f5657a7496c4e7b82d4244c01154242"
I20260619 18:47:47.112751 11408 raft_consensus.cc:3037] T b7b72c7308554519b2f323cae19b41c0 P 7c75efa0e0da41d7883d6fc4bc7c6f89 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:47:47.112844 11267 raft_consensus.cc:3037] T b7b72c7308554519b2f323cae19b41c0 P 3f5657a7496c4e7b82d4244c01154242 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:47:47.114732 11408 raft_consensus.cc:2454] T b7b72c7308554519b2f323cae19b41c0 P 7c75efa0e0da41d7883d6fc4bc7c6f89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69da4e79b53c43fc8becd20ad510c8b8 in term 1.
I20260619 18:47:47.114735 11267 raft_consensus.cc:2454] T b7b72c7308554519b2f323cae19b41c0 P 3f5657a7496c4e7b82d4244c01154242 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69da4e79b53c43fc8becd20ad510c8b8 in term 1.
I20260619 18:47:47.115084 11057 leader_election.cc:302] T b7b72c7308554519b2f323cae19b41c0 P 69da4e79b53c43fc8becd20ad510c8b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69da4e79b53c43fc8becd20ad510c8b8, 7c75efa0e0da41d7883d6fc4bc7c6f89; no voters:
I20260619 18:47:47.115345 11563 raft_consensus.cc:2781] T b7b72c7308554519b2f323cae19b41c0 P 69da4e79b53c43fc8becd20ad510c8b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:47:47.115761 11563 raft_consensus.cc:686] T b7b72c7308554519b2f323cae19b41c0 P 69da4e79b53c43fc8becd20ad510c8b8 [term 1 LEADER]: Becoming Leader. State: Replica: 69da4e79b53c43fc8becd20ad510c8b8, State: Running, Role: LEADER
I20260619 18:47:47.116040 11563 consensus_queue.cc:237] T b7b72c7308554519b2f323cae19b41c0 P 69da4e79b53c43fc8becd20ad510c8b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c75efa0e0da41d7883d6fc4bc7c6f89" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 36847 } } peers { permanent_uuid: "3f5657a7496c4e7b82d4244c01154242" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 37147 } } peers { permanent_uuid: "69da4e79b53c43fc8becd20ad510c8b8" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 34185 } }
I20260619 18:47:47.116726 11564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:47.116325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:41477 (local address 127.3.243.65:39269)
0619 18:47:47.116475 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:47.116566 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:41477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260619 18:47:47.118618 10988 catalog_manager.cc:5495] T b7b72c7308554519b2f323cae19b41c0 P 69da4e79b53c43fc8becd20ad510c8b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 69da4e79b53c43fc8becd20ad510c8b8 (127.3.243.65). New cstate: current_term: 1 leader_uuid: "69da4e79b53c43fc8becd20ad510c8b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c75efa0e0da41d7883d6fc4bc7c6f89" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 36847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3f5657a7496c4e7b82d4244c01154242" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 37147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69da4e79b53c43fc8becd20ad510c8b8" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 34185 } health_report { overall_health: HEALTHY } } }
I20260619 18:47:47.138226 11557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:47.134311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50212 (local address 127.3.243.65:34185)
0619 18:47:47.134457 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:47.134462 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:47.134493 (+ 31us) server_negotiation.cc:407] Connection header received
0619 18:47:47.134641 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:47.134644 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:47.134704 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:47.134772 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:47:47.135041 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:47.135754 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:47.136968 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:47.137360 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:47.137445 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:47.137722 (+ 277us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:47:47.137822 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:47:47.137886 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:47.137971 (+ 85us) server_negotiation.cc:299] Negotiation successful
0619 18:47:47.138027 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 18:47:47.140829 11267 raft_consensus.cc:1261] T b7b72c7308554519b2f323cae19b41c0 P 3f5657a7496c4e7b82d4244c01154242 [term 1 FOLLOWER]: Refusing update from remote peer 69da4e79b53c43fc8becd20ad510c8b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:47:47.140915 11408 raft_consensus.cc:1261] T b7b72c7308554519b2f323cae19b41c0 P 7c75efa0e0da41d7883d6fc4bc7c6f89 [term 1 FOLLOWER]: Refusing update from remote peer 69da4e79b53c43fc8becd20ad510c8b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:47:47.141412 11572 consensus_queue.cc:1035] T b7b72c7308554519b2f323cae19b41c0 P 69da4e79b53c43fc8becd20ad510c8b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f5657a7496c4e7b82d4244c01154242" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 37147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:47:47.141762 11563 consensus_queue.cc:1035] T b7b72c7308554519b2f323cae19b41c0 P 69da4e79b53c43fc8becd20ad510c8b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c75efa0e0da41d7883d6fc4bc7c6f89" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 36847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:47:47.143956 11569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:47.143474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:41477 (local address 127.3.243.66:45063)
0619 18:47:47.143641 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:47.143771 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:41477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260619 18:47:47.144024 11570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:47.143509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:41477 (local address 127.3.243.67:37443)
0619 18:47:47.143757 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:47.143850 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:41477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260619 18:47:47.153335 11564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:47.152887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:41477 (local address 127.3.243.65:53411)
0619 18:47:47.153046 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:47.153161 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:41477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260619 18:47:47.154551 11569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:47.153977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:41477 (local address 127.3.243.66:44103)
0619 18:47:47.154318 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:47.154404 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:41477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247}
I20260619 18:47:47.156958 11564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:47.156254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:41477 (local address 127.3.243.65:49619)
0619 18:47:47.156676 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:47.156782 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:41477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319}
I20260619 18:47:47.159402 11570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:47.155876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:41477 (local address 127.3.243.67:60269)
0619 18:47:47.159121 (+ 3245us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:47.159239 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:41477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3132}
I20260619 18:47:47.163470 10987 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39950:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:47:47.163714 10987 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39950:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:47:47.168388 10987 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ed237dce23ef407fa74b615f8f6c914f: Sending DeleteTablet for 3 replicas of tablet b7b72c7308554519b2f323cae19b41c0
I20260619 18:47:47.169313 11247 tablet_service.cc:1514] Processing DeleteTablet for tablet b7b72c7308554519b2f323cae19b41c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:47:47 UTC) from {username='oryx', principal='oryx/127.3.243.124@KRBTEST.COM'} at 127.0.0.1:50340
I20260619 18:47:47.169363 11388 tablet_service.cc:1514] Processing DeleteTablet for tablet b7b72c7308554519b2f323cae19b41c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:47:47 UTC) from {username='oryx', principal='oryx/127.3.243.124@KRBTEST.COM'} at 127.0.0.1:38776
I20260619 18:47:47.169364 11101 tablet_service.cc:1514] Processing DeleteTablet for tablet b7b72c7308554519b2f323cae19b41c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:47:47 UTC) from {username='oryx', principal='oryx/127.3.243.124@KRBTEST.COM'} at 127.0.0.1:50202
I20260619 18:47:47.171103 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 10960
I20260619 18:47:47.172618 11585 tablet_replica.cc:330] T b7b72c7308554519b2f323cae19b41c0 P 3f5657a7496c4e7b82d4244c01154242: stopping tablet replica
I20260619 18:47:47.172888 11585 raft_consensus.cc:2229] T b7b72c7308554519b2f323cae19b41c0 P 3f5657a7496c4e7b82d4244c01154242 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:47:47.173260 11585 raft_consensus.cc:2258] T b7b72c7308554519b2f323cae19b41c0 P 3f5657a7496c4e7b82d4244c01154242 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:47:47.173859 11584 tablet_replica.cc:330] T b7b72c7308554519b2f323cae19b41c0 P 7c75efa0e0da41d7883d6fc4bc7c6f89: stopping tablet replica
I20260619 18:47:47.174314 11586 tablet_replica.cc:330] T b7b72c7308554519b2f323cae19b41c0 P 69da4e79b53c43fc8becd20ad510c8b8: stopping tablet replica
I20260619 18:47:47.174544 11586 raft_consensus.cc:2229] T b7b72c7308554519b2f323cae19b41c0 P 69da4e79b53c43fc8becd20ad510c8b8 [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:47:47.174855 11585 ts_tablet_manager.cc:1905] T b7b72c7308554519b2f323cae19b41c0 P 3f5657a7496c4e7b82d4244c01154242: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:47:47.174965 11586 raft_consensus.cc:2258] T b7b72c7308554519b2f323cae19b41c0 P 69da4e79b53c43fc8becd20ad510c8b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:47:47.175275 11584 raft_consensus.cc:2229] T b7b72c7308554519b2f323cae19b41c0 P 7c75efa0e0da41d7883d6fc4bc7c6f89 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:47:47.175662 11584 raft_consensus.cc:2258] T b7b72c7308554519b2f323cae19b41c0 P 7c75efa0e0da41d7883d6fc4bc7c6f89 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:47:47.176867 11584 ts_tablet_manager.cc:1905] T b7b72c7308554519b2f323cae19b41c0 P 7c75efa0e0da41d7883d6fc4bc7c6f89: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:47:47.177218 11586 ts_tablet_manager.cc:1905] T b7b72c7308554519b2f323cae19b41c0 P 69da4e79b53c43fc8becd20ad510c8b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:47:47.179963 11585 ts_tablet_manager.cc:1918] T b7b72c7308554519b2f323cae19b41c0 P 3f5657a7496c4e7b82d4244c01154242: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:47:47.180075 11585 log.cc:1196] T b7b72c7308554519b2f323cae19b41c0 P 3f5657a7496c4e7b82d4244c01154242: Deleting WAL directory at /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/ts-1/wal/wals/b7b72c7308554519b2f323cae19b41c0
I20260619 18:47:47.180437 11585 ts_tablet_manager.cc:1939] T b7b72c7308554519b2f323cae19b41c0 P 3f5657a7496c4e7b82d4244c01154242: Deleting consensus metadata
I20260619 18:47:47.181331 10974 catalog_manager.cc:4841] TS 3f5657a7496c4e7b82d4244c01154242 (127.3.243.66:37147): tablet b7b72c7308554519b2f323cae19b41c0 (table test-table [id=3aa3d3cb858a42b3a09cc9edd65962a9]) successfully deleted
I20260619 18:47:47.183171 11584 ts_tablet_manager.cc:1918] T b7b72c7308554519b2f323cae19b41c0 P 7c75efa0e0da41d7883d6fc4bc7c6f89: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:47:47.183187 11586 ts_tablet_manager.cc:1918] T b7b72c7308554519b2f323cae19b41c0 P 69da4e79b53c43fc8becd20ad510c8b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:47:47.183301 11586 log.cc:1196] T b7b72c7308554519b2f323cae19b41c0 P 69da4e79b53c43fc8becd20ad510c8b8: Deleting WAL directory at /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/ts-0/wal/wals/b7b72c7308554519b2f323cae19b41c0
I20260619 18:47:47.183301 11584 log.cc:1196] T b7b72c7308554519b2f323cae19b41c0 P 7c75efa0e0da41d7883d6fc4bc7c6f89: Deleting WAL directory at /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894810715282-4045-0/minicluster-data/ts-2/wal/wals/b7b72c7308554519b2f323cae19b41c0
I20260619 18:47:47.183723 11586 ts_tablet_manager.cc:1939] T b7b72c7308554519b2f323cae19b41c0 P 69da4e79b53c43fc8becd20ad510c8b8: Deleting consensus metadata
I20260619 18:47:47.183723 11584 ts_tablet_manager.cc:1939] T b7b72c7308554519b2f323cae19b41c0 P 7c75efa0e0da41d7883d6fc4bc7c6f89: Deleting consensus metadata
I20260619 18:47:47.184734 10973 catalog_manager.cc:4841] TS 69da4e79b53c43fc8becd20ad510c8b8 (127.3.243.65:34185): tablet b7b72c7308554519b2f323cae19b41c0 (table test-table [id=3aa3d3cb858a42b3a09cc9edd65962a9]) successfully deleted
I20260619 18:47:47.184850 10975 catalog_manager.cc:4841] TS 7c75efa0e0da41d7883d6fc4bc7c6f89 (127.3.243.67:36847): tablet b7b72c7308554519b2f323cae19b41c0 (table test-table [id=3aa3d3cb858a42b3a09cc9edd65962a9]) successfully deleted
I20260619 18:47:47.194128 11557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:47.172322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50222 (local address 127.3.243.65:34185)
0619 18:47:47.172482 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:47.172487 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:47.174078 (+ 1591us) server_negotiation.cc:407] Connection header received
0619 18:47:47.174277 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:47.174284 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:47.174369 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:47.174456 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:47.184290 (+ 9834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:47.185166 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:47.186344 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:47.186734 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:47.186838 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:47.189171 (+ 2333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:47.189186 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:47.189201 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:47.189236 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:47.191525 (+ 2289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:47.192167 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:47.192177 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:47.192181 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:47.192322 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:47.192675 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:47.192680 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:47.192684 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:47.193026 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:47.193295 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:47.193622 (+ 327us) server_negotiation.cc:299] Negotiation successful
0619 18:47:47.193841 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 18:47:47.194749 11161 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:47:47.313575 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 10960
I20260619 18:47:47.320730 11541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:47.320352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:41477 (local address 127.0.0.1:33830)
0619 18:47:47.320498 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:47.320601 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:41477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260619 18:47:47.321617 11541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:47.321325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:41477 (local address 127.0.0.1:33846)
0619 18:47:47.321430 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:47.321495 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:41477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260619 18:47:47.333022 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 11178
I20260619 18:47:47.345777 11555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:47.333720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50356 (local address 127.3.243.66:37147)
0619 18:47:47.333895 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:47.333901 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:47.333922 (+ 21us) server_negotiation.cc:407] Connection header received
0619 18:47:47.334100 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:47.334105 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:47.334182 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:47.334266 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:47.335529 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:47.336439 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:47.337638 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:47.338047 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:47.338157 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:47.340523 (+ 2366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:47.340538 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:47.340555 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:47.340586 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:47.343137 (+ 2551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:47.343764 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:47.343771 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:47.343773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:47.343914 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:47.344262 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:47.344265 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:47.344267 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:47.344675 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:47.345036 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:47.345263 (+ 227us) server_negotiation.cc:299] Negotiation successful
0619 18:47:47.345449 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 18:47:47.346380 11307 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:47:47.437175 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 11178
I20260619 18:47:47.450935 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 11325
I20260619 18:47:47.463414 11553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:47.451754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38788 (local address 127.3.243.67:36847)
0619 18:47:47.451947 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:47.451953 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:47.451973 (+ 20us) server_negotiation.cc:407] Connection header received
0619 18:47:47.452100 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:47.452105 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:47.452205 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:47.452299 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:47.453610 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:47.454311 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:47.455636 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:47.456057 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:47.456336 (+ 279us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:47.458570 (+ 2234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:47.458586 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:47.458601 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:47.458643 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:47.461190 (+ 2547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:47.461788 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:47.461795 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:47.461798 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:47.461923 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:47.462200 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:47.462202 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:47.462204 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:47.462533 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:47.462738 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:47.462944 (+ 206us) server_negotiation.cc:299] Negotiation successful
0619 18:47:47.463103 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 18:47:47.464016 11448 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:47:47.555392 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 11325
I20260619 18:47:47.568740 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 10661
I20260619 18:47:47.569808 10827 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:47:47.686285 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 10661
I20260619 18:47:47.702641 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 10763
I20260619 18:47:47.703644 11018 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:47:47.823320 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 10763
2026-06-19T18:47:47Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (11056 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:47:47 dist-test-slave-m9wg krb5kdc[11595](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:47:47 dist-test-slave-m9wg krb5kdc[11595](info): set up 2 sockets
Jun 19 18:47:47 dist-test-slave-m9wg krb5kdc[11595](info): commencing operation
krb5kdc: starting...
W20260619 18:47:48.941190 4045 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.005s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/krb5kdc/test-user.keytab.
Jun 19 18:47:49 dist-test-slave-m9wg krb5kdc[11595](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894869, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:47:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:47:49Z Disabled control of system clock
WARNING: no policy specified for oryx/127.3.243.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.243.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.243.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/krb5kdc/oryx_127.3.243.126.keytab.
Entry for principal oryx/127.3.243.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/krb5kdc/oryx_127.3.243.126.keytab.
I20260619 18:47:49.133237 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.126:41539
--webserver_interface=127.3.243.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/krb5kdc/oryx_127.3.243.126.keytab
--principal=oryx/127.3.243.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:39313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.126:41539
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:47:49.443075 11608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:47:49.443396 11608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:47:49.443490 11608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:47:49.450842 11608 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:47:49.450963 11608 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 18:47:49.451011 11608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:47:49.451071 11608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:47:49.451123 11608 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:47:49.451182 11608 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:47:49.461217 11608 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:39313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.243.126:41539
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/krb5kdc/oryx_127.3.243.126.keytab
--principal=oryx/127.3.243.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.126:41539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.243.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:47:49.462467 11608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:47:49.463724 11608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:47:49.475670 11616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:49.475773 11614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:49.475759 11613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:47:49.481508 11608 server_base.cc:1034] running on GCE node
I20260619 18:47:49.482291 11608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:47:49.483774 11608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:47:49.484997 11608 hybrid_clock.cc:648] HybridClock initialized: now 1781894869484932 us; error 29 us; skew 500 ppm
I20260619 18:47:49.485313 11608 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:47:49 dist-test-slave-m9wg krb5kdc[11595](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894869, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.243.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:47:49.492293 11608 init.cc:377] Logged in from keytab as oryx/127.3.243.126@KRBTEST.COM (short username oryx)
I20260619 18:47:49.493754 11608 webserver.cc:460] Webserver started at http://127.3.243.126:36315/ using document root <none> and password file <none>
I20260619 18:47:49.494467 11608 fs_manager.cc:359] Metadata directory not provided
I20260619 18:47:49.494565 11608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:47:49.494863 11608 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:47:49.498171 11608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/minicluster-data/master-0/data/instance:
uuid: "022c3c6be95b46f8bf8e526a33780f6d"
format_stamp: "Formatted at 2026-06-19 18:47:49 on dist-test-slave-m9wg"
server_key: "acff3958c970a44a012599924baa0605"
server_key_iv: "5b22b45f4b40ddaec72fb28a0226bda1"
server_key_version: "encryptionkey@0"
I20260619 18:47:49.498956 11608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/minicluster-data/master-0/wal/instance:
uuid: "022c3c6be95b46f8bf8e526a33780f6d"
format_stamp: "Formatted at 2026-06-19 18:47:49 on dist-test-slave-m9wg"
server_key: "acff3958c970a44a012599924baa0605"
server_key_iv: "5b22b45f4b40ddaec72fb28a0226bda1"
server_key_version: "encryptionkey@0"
I20260619 18:47:49.503926 11608 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 18:47:49.507305 11623 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:49.508752 11608 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:47:49.508919 11608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/minicluster-data/master-0/wal
uuid: "022c3c6be95b46f8bf8e526a33780f6d"
format_stamp: "Formatted at 2026-06-19 18:47:49 on dist-test-slave-m9wg"
server_key: "acff3958c970a44a012599924baa0605"
server_key_iv: "5b22b45f4b40ddaec72fb28a0226bda1"
server_key_version: "encryptionkey@0"
I20260619 18:47:49.509146 11608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:47:49.523145 11608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:47:49.526721 11608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:47:49.526924 11608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:47:49.546710 11608 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.126:41539
I20260619 18:47:49.546729 11684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.126:41539 every 8 connection(s)
I20260619 18:47:49.548107 11608 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
I20260619 18:47:49.550137 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 11608
I20260619 18:47:49.550333 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/minicluster-data/master-0/wal/instance
I20260619 18:47:49.551133 4045 external_mini_cluster.cc:1428] Setting key 86d51372e35a8e602b0fb3b861802c2f
I20260619 18:47:49.552469 11685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 18:47:49 dist-test-slave-m9wg krb5kdc[11595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894869, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM
I20260619 18:47:49.566040 11685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 022c3c6be95b46f8bf8e526a33780f6d: Bootstrap starting.
I20260619 18:47:49.569387 11685 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 022c3c6be95b46f8bf8e526a33780f6d: Neither blocks nor log segments found. Creating new log.
I20260619 18:47:49.570619 11685 log.cc:824] T 00000000000000000000000000000000 P 022c3c6be95b46f8bf8e526a33780f6d: Log is configured to *not* fsync() on all Append() calls
I20260619 18:47:49.573550 11688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:49.553135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54990 (local address 127.3.243.126:41539)
0619 18:47:49.553874 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:49.553885 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:49.553923 (+ 38us) server_negotiation.cc:407] Connection header received
0619 18:47:49.555463 (+ 1540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:49.555483 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:49.555982 (+ 499us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:49.556351 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:49.558043 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:49.559381 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:49.561061 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:49.561765 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:49.561942 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:49.566514 (+ 4572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:49.566521 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:49.566535 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:49.566589 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:49.569402 (+ 2813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:49.570199 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:49.570205 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:49.570212 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:49.570406 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:49.570855 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:49.570860 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:49.570862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:49.571573 (+ 711us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:49.571872 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:49.572237 (+ 365us) server_negotiation.cc:299] Negotiation successful
0619 18:47:49.572613 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":357,"threads_started":1}
I20260619 18:47:49.573618 11685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 022c3c6be95b46f8bf8e526a33780f6d: No bootstrap required, opened a new log
I20260619 18:47:49.581269 11685 raft_consensus.cc:348] T 00000000000000000000000000000000 P 022c3c6be95b46f8bf8e526a33780f6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "022c3c6be95b46f8bf8e526a33780f6d" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 41539 } }
I20260619 18:47:49.581688 11685 raft_consensus.cc:374] T 00000000000000000000000000000000 P 022c3c6be95b46f8bf8e526a33780f6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:47:49.581777 11685 raft_consensus.cc:729] T 00000000000000000000000000000000 P 022c3c6be95b46f8bf8e526a33780f6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 022c3c6be95b46f8bf8e526a33780f6d, State: Initialized, Role: FOLLOWER
I20260619 18:47:49.582295 11685 consensus_queue.cc:260] T 00000000000000000000000000000000 P 022c3c6be95b46f8bf8e526a33780f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "022c3c6be95b46f8bf8e526a33780f6d" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 41539 } }
I20260619 18:47:49.582502 11685 raft_consensus.cc:388] T 00000000000000000000000000000000 P 022c3c6be95b46f8bf8e526a33780f6d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:47:49.582634 11685 raft_consensus.cc:482] T 00000000000000000000000000000000 P 022c3c6be95b46f8bf8e526a33780f6d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:47:49.582796 11685 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 022c3c6be95b46f8bf8e526a33780f6d [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:47:49.585227 11685 raft_consensus.cc:504] T 00000000000000000000000000000000 P 022c3c6be95b46f8bf8e526a33780f6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "022c3c6be95b46f8bf8e526a33780f6d" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 41539 } }
I20260619 18:47:49.585770 11685 leader_election.cc:302] T 00000000000000000000000000000000 P 022c3c6be95b46f8bf8e526a33780f6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 022c3c6be95b46f8bf8e526a33780f6d; no voters:
I20260619 18:47:49.586336 11685 leader_election.cc:288] T 00000000000000000000000000000000 P 022c3c6be95b46f8bf8e526a33780f6d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:47:49.586455 11690 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 022c3c6be95b46f8bf8e526a33780f6d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:47:49.587121 11690 raft_consensus.cc:686] T 00000000000000000000000000000000 P 022c3c6be95b46f8bf8e526a33780f6d [term 1 LEADER]: Becoming Leader. State: Replica: 022c3c6be95b46f8bf8e526a33780f6d, State: Running, Role: LEADER
I20260619 18:47:49.587667 11690 consensus_queue.cc:237] T 00000000000000000000000000000000 P 022c3c6be95b46f8bf8e526a33780f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "022c3c6be95b46f8bf8e526a33780f6d" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 41539 } }
I20260619 18:47:49.588294 11685 sys_catalog.cc:564] T 00000000000000000000000000000000 P 022c3c6be95b46f8bf8e526a33780f6d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:47:49.590247 11692 sys_catalog.cc:455] T 00000000000000000000000000000000 P 022c3c6be95b46f8bf8e526a33780f6d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 022c3c6be95b46f8bf8e526a33780f6d. Latest consensus state: current_term: 1 leader_uuid: "022c3c6be95b46f8bf8e526a33780f6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "022c3c6be95b46f8bf8e526a33780f6d" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 41539 } } }
I20260619 18:47:49.591158 11692 sys_catalog.cc:458] T 00000000000000000000000000000000 P 022c3c6be95b46f8bf8e526a33780f6d [sys.catalog]: This master's current role is: LEADER
I20260619 18:47:49.590716 11691 sys_catalog.cc:455] T 00000000000000000000000000000000 P 022c3c6be95b46f8bf8e526a33780f6d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "022c3c6be95b46f8bf8e526a33780f6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "022c3c6be95b46f8bf8e526a33780f6d" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 41539 } } }
I20260619 18:47:49.591375 11691 sys_catalog.cc:458] T 00000000000000000000000000000000 P 022c3c6be95b46f8bf8e526a33780f6d [sys.catalog]: This master's current role is: LEADER
I20260619 18:47:49.596901 11697 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:47:49.601903 11697 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:47:49.609889 11697 catalog_manager.cc:1347] Generated new cluster ID: 3506784150934175bc070b5893a6723e
I20260619 18:47:49.610009 11697 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:47:49.626609 11697 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:47:49.627830 11697 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:47:49.636193 11697 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 022c3c6be95b46f8bf8e526a33780f6d: Generated new TSK 0
I20260619 18:47:49.637563 11697 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.3.243.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.243.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.243.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/krb5kdc/oryx_127.3.243.65.keytab.
Entry for principal oryx/127.3.243.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/krb5kdc/oryx_127.3.243.65.keytab.
I20260619 18:47:49.699321 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.65:0
--local_ip_for_outbound_sockets=127.3.243.65
--webserver_interface=127.3.243.65
--webserver_port=0
--tserver_master_addrs=127.3.243.126:41539
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/krb5kdc/oryx_127.3.243.65.keytab
--principal=oryx/127.3.243.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:39313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:47:50.011440 11711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:47:50.011709 11711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:47:50.011806 11711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:47:50.019233 11711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:47:50.019336 11711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:47:50.019426 11711 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 18:47:50.019507 11711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.65
I20260619 18:47:50.029001 11711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:39313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/krb5kdc/oryx_127.3.243.65.keytab
--principal=oryx/127.3.243.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.243.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.243.126:41539
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.65
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:47:50.030138 11711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:47:50.031374 11711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:47:50.043540 11716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:50.043576 11719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:50.043493 11717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:47:50.044413 11711 server_base.cc:1034] running on GCE node
I20260619 18:47:50.045003 11711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:47:50.045835 11711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:47:50.047040 11711 hybrid_clock.cc:648] HybridClock initialized: now 1781894870046969 us; error 44 us; skew 500 ppm
I20260619 18:47:50.047314 11711 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:47:50 dist-test-slave-m9wg krb5kdc[11595](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894870, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.243.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:47:50.050303 11711 init.cc:377] Logged in from keytab as oryx/127.3.243.65@KRBTEST.COM (short username oryx)
I20260619 18:47:50.051549 11711 webserver.cc:460] Webserver started at http://127.3.243.65:40103/ using document root <none> and password file <none>
I20260619 18:47:50.052232 11711 fs_manager.cc:359] Metadata directory not provided
I20260619 18:47:50.052330 11711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:47:50.052623 11711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:47:50.055299 11711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/minicluster-data/ts-0/data/instance:
uuid: "653a69448f244a2a99faad646667dda7"
format_stamp: "Formatted at 2026-06-19 18:47:50 on dist-test-slave-m9wg"
server_key: "9d931d2148e6433859da52d2894ef759"
server_key_iv: "968b4a9d2e2636a65694a90baaf8c883"
server_key_version: "encryptionkey@0"
I20260619 18:47:50.056066 11711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/minicluster-data/ts-0/wal/instance:
uuid: "653a69448f244a2a99faad646667dda7"
format_stamp: "Formatted at 2026-06-19 18:47:50 on dist-test-slave-m9wg"
server_key: "9d931d2148e6433859da52d2894ef759"
server_key_iv: "968b4a9d2e2636a65694a90baaf8c883"
server_key_version: "encryptionkey@0"
I20260619 18:47:50.060822 11711 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 18:47:50.064028 11726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:50.065281 11711 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 18:47:50.065421 11711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/minicluster-data/ts-0/wal
uuid: "653a69448f244a2a99faad646667dda7"
format_stamp: "Formatted at 2026-06-19 18:47:50 on dist-test-slave-m9wg"
server_key: "9d931d2148e6433859da52d2894ef759"
server_key_iv: "968b4a9d2e2636a65694a90baaf8c883"
server_key_version: "encryptionkey@0"
I20260619 18:47:50.065613 11711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:47:50.080811 11711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:47:50.084398 11711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:47:50.084637 11711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:47:50.086990 11711 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:47:50.087073 11711 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:50.087208 11711 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:47:50.087265 11711 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 18:47:50 dist-test-slave-m9wg krb5kdc[11595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894870, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.243.65@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM
I20260619 18:47:50.108319 11735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:50.087600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.65:59979 (local address 127.3.243.126:41539)
0619 18:47:50.088211 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:50.088219 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:50.089232 (+ 1013us) server_negotiation.cc:407] Connection header received
0619 18:47:50.091382 (+ 2150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:50.091387 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:50.091473 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:50.091580 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:50.093842 (+ 2262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:50.095081 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:50.096912 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:50.097458 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:50.097570 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:50.102028 (+ 4458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:50.102033 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:50.102036 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:50.102069 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:50.105119 (+ 3050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:50.106010 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:50.106014 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:50.106017 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:50.106133 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:50.106571 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:50.106574 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:50.106576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:50.106834 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:50.106938 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:50.107828 (+ 890us) server_negotiation.cc:299] Negotiation successful
0619 18:47:50.107988 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":291,"threads_started":1}
I20260619 18:47:50.110148 11736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:50.088214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:41539 (local address 127.3.243.65:59979)
0619 18:47:50.089065 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:50.089117 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:50.091114 (+ 1997us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:50.092006 (+ 892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:50.092014 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:50.092844 (+ 830us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:47:50.093652 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:50.093675 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:50.095241 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:50.095245 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:50.096723 (+ 1478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:50.096739 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:50.097686 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:50.097694 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:50.097920 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:50.098598 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:47:50.098640 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:47:50.101762 (+ 3122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:47:50.105267 (+ 3505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:50.105277 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:50.105295 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:50.105895 (+ 600us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:50.106221 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:50.106226 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:50.106230 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:50.106479 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:50.106938 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:47:50.106946 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:47:50.107437 (+ 491us) client_negotiation.cc:769] Sending connection context
0619 18:47:50.107740 (+ 303us) client_negotiation.cc:240] Negotiation successful
0619 18:47:50.108335 (+ 595us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":362,"threads_started":1}
I20260619 18:47:50.129833 11735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:50.116398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.65:49615 (local address 127.3.243.126:41539)
0619 18:47:50.116569 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:50.116574 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:50.117158 (+ 584us) server_negotiation.cc:407] Connection header received
0619 18:47:50.117322 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:50.117326 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:50.117384 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:50.117458 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:50.118612 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:50.119288 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:50.121294 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:50.121706 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:50.121801 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:50.124179 (+ 2378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:50.124185 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:50.124190 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:50.124223 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:50.126792 (+ 2569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:50.127680 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:50.127685 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:50.127688 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:50.127845 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:50.128323 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:50.128328 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:50.128330 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:50.128671 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:50.128834 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:50.129294 (+ 460us) server_negotiation.cc:299] Negotiation successful
0619 18:47:50.129499 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 18:47:50.129952 11804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:50.116417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:41539 (local address 127.3.243.65:49615)
0619 18:47:50.117037 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:50.117059 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:50.117217 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:50.117498 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:50.117504 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:50.118091 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:47:50.118483 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:50.118496 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:50.119403 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:50.119408 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:50.121141 (+ 1733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:50.121160 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:50.121877 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:50.121884 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:50.122085 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:50.122728 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:47:50.122753 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:47:50.123965 (+ 1212us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:47:50.126915 (+ 2950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:50.126921 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:50.126925 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:50.127553 (+ 628us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:50.127967 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:50.127973 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:50.127977 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:50.128144 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:50.128816 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:47:50.128822 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:47:50.129040 (+ 218us) client_negotiation.cc:769] Sending connection context
0619 18:47:50.129178 (+ 138us) client_negotiation.cc:240] Negotiation successful
0619 18:47:50.129516 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":313,"threads_started":1}
I20260619 18:47:50.136771 11711 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.65:35771
I20260619 18:47:50.136809 11846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.65:35771 every 8 connection(s)
I20260619 18:47:50.137892 11711 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/minicluster-data/ts-0/data/info.pb
I20260619 18:47:50.138209 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 11711
I20260619 18:47:50.138388 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/minicluster-data/ts-0/wal/instance
I20260619 18:47:50.139178 4045 external_mini_cluster.cc:1428] Setting key b7b9370b62cc691273f078f8a364dd73
I20260619 18:47:50.139904 11847 heartbeater.cc:347] Connected to a master server at 127.3.243.126:41539
I20260619 18:47:50.140087 11847 heartbeater.cc:464] Registering TS with master...
I20260619 18:47:50.140698 11847 heartbeater.cc:511] Master 127.3.243.126:41539 requested a full tablet report, sending...
I20260619 18:47:50.142545 11640 ts_manager.cc:194] Registered new tserver with Master: 653a69448f244a2a99faad646667dda7 (127.3.243.65:35771)
I20260619 18:47:50.143925 11640 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.243.65@KRBTEST.COM'} at 127.3.243.65:59979
WARNING: no policy specified for oryx/127.3.243.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.243.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.243.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/krb5kdc/oryx_127.3.243.66.keytab.
Entry for principal oryx/127.3.243.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/krb5kdc/oryx_127.3.243.66.keytab.
I20260619 18:47:50.181481 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.66:0
--local_ip_for_outbound_sockets=127.3.243.66
--webserver_interface=127.3.243.66
--webserver_port=0
--tserver_master_addrs=127.3.243.126:41539
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/krb5kdc/oryx_127.3.243.66.keytab
--principal=oryx/127.3.243.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:39313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:47:50.489751 11851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:47:50.490027 11851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:47:50.490180 11851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:47:50.497388 11851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:47:50.497506 11851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:47:50.497634 11851 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 18:47:50.497746 11851 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.66
I20260619 18:47:50.507330 11851 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:39313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/krb5kdc/oryx_127.3.243.66.keytab
--principal=oryx/127.3.243.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.243.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.243.126:41539
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.66
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:47:50.508613 11851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:47:50.509927 11851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:47:50.521981 11857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:50.522037 11859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:50.522068 11856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:47:50.523330 11851 server_base.cc:1034] running on GCE node
I20260619 18:47:50.523869 11851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:47:50.524748 11851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:47:50.525949 11851 hybrid_clock.cc:648] HybridClock initialized: now 1781894870525882 us; error 48 us; skew 500 ppm
I20260619 18:47:50.526238 11851 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:47:50 dist-test-slave-m9wg krb5kdc[11595](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894870, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.243.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:47:50.529088 11851 init.cc:377] Logged in from keytab as oryx/127.3.243.66@KRBTEST.COM (short username oryx)
I20260619 18:47:50.530400 11851 webserver.cc:460] Webserver started at http://127.3.243.66:37307/ using document root <none> and password file <none>
I20260619 18:47:50.531141 11851 fs_manager.cc:359] Metadata directory not provided
I20260619 18:47:50.531236 11851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:47:50.531526 11851 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:47:50.534325 11851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/minicluster-data/ts-1/data/instance:
uuid: "9ceae0288d68442b9b808ed536256768"
format_stamp: "Formatted at 2026-06-19 18:47:50 on dist-test-slave-m9wg"
server_key: "bd4f7241a76ac01ce546f02a24c3c33a"
server_key_iv: "bdd61a416355fd8eb0dc529b662db667"
server_key_version: "encryptionkey@0"
I20260619 18:47:50.535074 11851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/minicluster-data/ts-1/wal/instance:
uuid: "9ceae0288d68442b9b808ed536256768"
format_stamp: "Formatted at 2026-06-19 18:47:50 on dist-test-slave-m9wg"
server_key: "bd4f7241a76ac01ce546f02a24c3c33a"
server_key_iv: "bdd61a416355fd8eb0dc529b662db667"
server_key_version: "encryptionkey@0"
I20260619 18:47:50.539958 11851 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 18:47:50.543263 11866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:50.544539 11851 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:47:50.544674 11851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/minicluster-data/ts-1/wal
uuid: "9ceae0288d68442b9b808ed536256768"
format_stamp: "Formatted at 2026-06-19 18:47:50 on dist-test-slave-m9wg"
server_key: "bd4f7241a76ac01ce546f02a24c3c33a"
server_key_iv: "bdd61a416355fd8eb0dc529b662db667"
server_key_version: "encryptionkey@0"
I20260619 18:47:50.544862 11851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:47:50.580976 11851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:47:50.584765 11851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:47:50.585073 11851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:47:50.587559 11851 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:47:50.587644 11851 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:50.587777 11851 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:47:50.587831 11851 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 18:47:50 dist-test-slave-m9wg krb5kdc[11595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894870, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.243.66@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM
I20260619 18:47:50.610450 11735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:50.588248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.66:43829 (local address 127.3.243.126:41539)
0619 18:47:50.588418 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:50.588424 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:50.589888 (+ 1464us) server_negotiation.cc:407] Connection header received
0619 18:47:50.592871 (+ 2983us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:50.592877 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:50.592965 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:50.593061 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:50.595083 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:50.596304 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:50.598402 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:50.598928 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:50.599039 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:50.603404 (+ 4365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:50.603411 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:50.603416 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:50.603451 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:50.606881 (+ 3430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:50.607768 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:50.607773 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:50.607777 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:50.607949 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:50.608460 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:50.608465 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:50.608468 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:50.608849 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:50.609004 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:50.609879 (+ 875us) server_negotiation.cc:299] Negotiation successful
0619 18:47:50.610094 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 18:47:50.611581 11875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:50.588867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:41539 (local address 127.3.243.66:43829)
0619 18:47:50.589721 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:50.589773 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:50.592386 (+ 2613us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:50.593360 (+ 974us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:50.593368 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:50.594059 (+ 691us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:47:50.594882 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:50.594909 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:50.596470 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:50.596475 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:50.598269 (+ 1794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:50.598284 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:50.599181 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:50.599189 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:50.599457 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:50.600222 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:47:50.600256 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:47:50.603201 (+ 2945us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:47:50.607009 (+ 3808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:50.607018 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:50.607040 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:50.607634 (+ 594us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:50.608052 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:50.608055 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:50.608057 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:50.608344 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:50.609001 (+ 657us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:47:50.609008 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:47:50.609449 (+ 441us) client_negotiation.cc:769] Sending connection context
0619 18:47:50.609781 (+ 332us) client_negotiation.cc:240] Negotiation successful
0619 18:47:50.610340 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":338,"threads_started":1}
I20260619 18:47:50.629721 11851 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.66:45059
I20260619 18:47:50.629760 11985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.66:45059 every 8 connection(s)
I20260619 18:47:50.630579 11958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:50.617719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:41539 (local address 127.3.243.66:32937)
0619 18:47:50.618319 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:50.618350 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:50.618590 (+ 240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:50.619022 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:50.619028 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:50.619437 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:47:50.619707 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:50.619716 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:50.620749 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:50.620753 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:50.622274 (+ 1521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:50.622292 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:50.623129 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:50.623136 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:50.623284 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:50.623834 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:47:50.623851 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:47:50.624743 (+ 892us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:47:50.628042 (+ 3299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:50.628047 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:50.628050 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:50.628467 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:50.628849 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:50.628854 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:50.628858 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:50.629028 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:50.629691 (+ 663us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:47:50.629695 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:47:50.629883 (+ 188us) client_negotiation.cc:769] Sending connection context
0619 18:47:50.630009 (+ 126us) client_negotiation.cc:240] Negotiation successful
0619 18:47:50.630268 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":305,"threads_started":1}
I20260619 18:47:50.631227 11735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:50.617719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.66:32937 (local address 127.3.243.126:41539)
0619 18:47:50.617910 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:50.617916 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:50.618458 (+ 542us) server_negotiation.cc:407] Connection header received
0619 18:47:50.618773 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:50.618779 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:50.618866 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:50.618959 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:50.619836 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:50.620602 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:50.622403 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:50.622933 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:50.623040 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:50.624943 (+ 1903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:50.624947 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:50.624951 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:50.624979 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:50.627893 (+ 2914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:50.628575 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:50.628579 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:50.628583 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:50.628747 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:50.629188 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:50.629194 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:50.629198 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:50.629573 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:50.629704 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:50.630856 (+ 1152us) server_negotiation.cc:299] Negotiation successful
0619 18:47:50.631024 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 18:47:50.631292 11851 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/minicluster-data/ts-1/data/info.pb
I20260619 18:47:50.633419 11986 heartbeater.cc:347] Connected to a master server at 127.3.243.126:41539
I20260619 18:47:50.633679 11986 heartbeater.cc:464] Registering TS with master...
I20260619 18:47:50.634773 11986 heartbeater.cc:511] Master 127.3.243.126:41539 requested a full tablet report, sending...
I20260619 18:47:50.636278 11640 ts_manager.cc:194] Registered new tserver with Master: 9ceae0288d68442b9b808ed536256768 (127.3.243.66:45059)
I20260619 18:47:50.637399 11640 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.243.66@KRBTEST.COM'} at 127.3.243.66:43829
I20260619 18:47:50.639104 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 11851
I20260619 18:47:50.639283 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/minicluster-data/ts-1/wal/instance
I20260619 18:47:50.639704 4045 external_mini_cluster.cc:1428] Setting key 9765586b8d40ea36cf6cda000ee9e910
WARNING: no policy specified for oryx/127.3.243.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.243.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.243.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/krb5kdc/oryx_127.3.243.67.keytab.
Entry for principal oryx/127.3.243.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/krb5kdc/oryx_127.3.243.67.keytab.
I20260619 18:47:50.682430 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.67:0
--local_ip_for_outbound_sockets=127.3.243.67
--webserver_interface=127.3.243.67
--webserver_port=0
--tserver_master_addrs=127.3.243.126:41539
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/krb5kdc/oryx_127.3.243.67.keytab
--principal=oryx/127.3.243.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:39313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:47:50.995280 11990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:47:50.995556 11990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:47:50.995702 11990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:47:51.002748 11990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:47:51.002862 11990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:47:51.002985 11990 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 18:47:51.003098 11990 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.67
I20260619 18:47:51.012830 11990 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:39313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/krb5kdc/oryx_127.3.243.67.keytab
--principal=oryx/127.3.243.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.243.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.243.126:41539
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.67
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:47:51.014144 11990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:47:51.015486 11990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:47:51.027732 11995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:51.027782 11996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:51.027760 11998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:47:51.028916 11990 server_base.cc:1034] running on GCE node
I20260619 18:47:51.029471 11990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:47:51.030359 11990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:47:51.031558 11990 hybrid_clock.cc:648] HybridClock initialized: now 1781894871031483 us; error 55 us; skew 500 ppm
I20260619 18:47:51.031828 11990 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:47:51 dist-test-slave-m9wg krb5kdc[11595](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894871, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.243.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:47:51.034771 11990 init.cc:377] Logged in from keytab as oryx/127.3.243.67@KRBTEST.COM (short username oryx)
I20260619 18:47:51.036098 11990 webserver.cc:460] Webserver started at http://127.3.243.67:34749/ using document root <none> and password file <none>
I20260619 18:47:51.036773 11990 fs_manager.cc:359] Metadata directory not provided
I20260619 18:47:51.036856 11990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:47:51.037107 11990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:47:51.039824 11990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/minicluster-data/ts-2/data/instance:
uuid: "a236998d0ba646d693900cf720e416a1"
format_stamp: "Formatted at 2026-06-19 18:47:51 on dist-test-slave-m9wg"
server_key: "01fb86582c5e4b05183aaafdace6b29d"
server_key_iv: "7f5c7e162147e743830921dfb0b9a117"
server_key_version: "encryptionkey@0"
I20260619 18:47:51.040594 11990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/minicluster-data/ts-2/wal/instance:
uuid: "a236998d0ba646d693900cf720e416a1"
format_stamp: "Formatted at 2026-06-19 18:47:51 on dist-test-slave-m9wg"
server_key: "01fb86582c5e4b05183aaafdace6b29d"
server_key_iv: "7f5c7e162147e743830921dfb0b9a117"
server_key_version: "encryptionkey@0"
I20260619 18:47:51.045774 11990 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 18:47:51.048972 12005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:51.050294 11990 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260619 18:47:51.050439 11990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/minicluster-data/ts-2/wal
uuid: "a236998d0ba646d693900cf720e416a1"
format_stamp: "Formatted at 2026-06-19 18:47:51 on dist-test-slave-m9wg"
server_key: "01fb86582c5e4b05183aaafdace6b29d"
server_key_iv: "7f5c7e162147e743830921dfb0b9a117"
server_key_version: "encryptionkey@0"
I20260619 18:47:51.050637 11990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:47:51.067391 11990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:47:51.071573 11990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:47:51.071810 11990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:47:51.074330 11990 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:47:51.074412 11990 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:51.074558 11990 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:47:51.074620 11990 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 18:47:51 dist-test-slave-m9wg krb5kdc[11595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894871, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.243.67@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM
I20260619 18:47:51.098502 12014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:51.075525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:41539 (local address 127.3.243.67:40885)
0619 18:47:51.076385 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:51.076438 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:51.078280 (+ 1842us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:51.079007 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:51.079020 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:51.079762 (+ 742us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:47:51.080570 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:51.080606 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:51.082184 (+ 1578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:51.082190 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:51.084184 (+ 1994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:51.084199 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:51.085324 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:51.085330 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:51.085578 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:51.086276 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:47:51.086312 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:47:51.089725 (+ 3413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:47:51.093414 (+ 3689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:51.093424 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:51.093440 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:51.094078 (+ 638us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:51.094531 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:51.094539 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:51.094543 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:51.094850 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:51.095542 (+ 692us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:47:51.095548 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:47:51.096053 (+ 505us) client_negotiation.cc:769] Sending connection context
0619 18:47:51.096418 (+ 365us) client_negotiation.cc:240] Negotiation successful
0619 18:47:51.096911 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":354,"threads_started":1}
I20260619 18:47:51.099622 11735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:51.074998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.67:40885 (local address 127.3.243.126:41539)
0619 18:47:51.075183 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:51.075189 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:51.076560 (+ 1371us) server_negotiation.cc:407] Connection header received
0619 18:47:51.078548 (+ 1988us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:51.078554 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:51.078652 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:51.078737 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:51.080822 (+ 2085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:51.082008 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:51.084315 (+ 2307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:51.084903 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:51.085026 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:51.090009 (+ 4983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:51.090016 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:51.090021 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:51.090065 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:51.093248 (+ 3183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:51.094247 (+ 999us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:51.094253 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:51.094257 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:51.094426 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:51.094966 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:51.094972 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:51.094975 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:51.095337 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:51.095479 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:51.099087 (+ 3608us) server_negotiation.cc:299] Negotiation successful
0619 18:47:51.099354 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 18:47:51.118525 12099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:51.105494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:41539 (local address 127.3.243.67:42361)
0619 18:47:51.106073 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:51.106095 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:51.106212 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:51.106507 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:51.106510 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:51.106898 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:47:51.107175 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:51.107184 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:51.108621 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:51.108627 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:51.109869 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:51.109888 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:51.110717 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:51.110723 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:51.110941 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:51.111652 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:47:51.111679 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:47:51.112709 (+ 1030us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:47:51.116050 (+ 3341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:51.116056 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:51.116060 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:51.116557 (+ 497us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:51.116968 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:51.116971 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:51.116973 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:51.117083 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:51.117631 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:47:51.117635 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:47:51.117823 (+ 188us) client_negotiation.cc:769] Sending connection context
0619 18:47:51.117963 (+ 140us) client_negotiation.cc:240] Negotiation successful
0619 18:47:51.118201 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":305,"threads_started":1}
I20260619 18:47:51.118662 11735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:51.105472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.67:42361 (local address 127.3.243.126:41539)
0619 18:47:51.105653 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:51.105660 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:51.106173 (+ 513us) server_negotiation.cc:407] Connection header received
0619 18:47:51.106319 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:51.106325 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:51.106404 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:51.106499 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:51.107288 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:51.108432 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:51.110011 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:51.110513 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:51.110642 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:51.112908 (+ 2266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:51.112914 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:51.112919 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:51.112956 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:51.115925 (+ 2969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:51.116696 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:51.116702 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:51.116706 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:51.116869 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:51.117169 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:51.117173 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:51.117177 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:51.117514 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:51.117661 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:51.118078 (+ 417us) server_negotiation.cc:299] Negotiation successful
0619 18:47:51.118312 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 18:47:51.120828 11990 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.67:40189
I20260619 18:47:51.120847 12124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.67:40189 every 8 connection(s)
I20260619 18:47:51.122113 11990 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/minicluster-data/ts-2/data/info.pb
I20260619 18:47:51.123982 12125 heartbeater.cc:347] Connected to a master server at 127.3.243.126:41539
I20260619 18:47:51.124197 12125 heartbeater.cc:464] Registering TS with master...
I20260619 18:47:51.124776 12125 heartbeater.cc:511] Master 127.3.243.126:41539 requested a full tablet report, sending...
I20260619 18:47:51.125941 11640 ts_manager.cc:194] Registered new tserver with Master: a236998d0ba646d693900cf720e416a1 (127.3.243.67:40189)
I20260619 18:47:51.126691 11640 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.243.67@KRBTEST.COM'} at 127.3.243.67:40885
I20260619 18:47:51.131914 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 11990
I20260619 18:47:51.132041 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/minicluster-data/ts-2/wal/instance
I20260619 18:47:51.132478 4045 external_mini_cluster.cc:1428] Setting key 2bd1ac720674612f321080d786cc98b7
I20260619 18:47:51.135056 4045 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 18:47:51.146898 11847 heartbeater.cc:503] Master 127.3.243.126:41539 was elected leader, sending a full tablet report...
Jun 19 18:47:51 dist-test-slave-m9wg krb5kdc[11595](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894871, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 18:47:51 dist-test-slave-m9wg krb5kdc[11595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.3.243.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:47:51 dist-test-slave-m9wg krb5kdc[11595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.3.243.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:47:51.639801 11986 heartbeater.cc:503] Master 127.3.243.126:41539 was elected leader, sending a full tablet report...
I20260619 18:47:52.128993 12125 heartbeater.cc:503] Master 127.3.243.126:41539 was elected leader, sending a full tablet report...
W20260619 18:47:52.143024 12133 sasl_common.cc:323] Time spent calling sasl_client_start(): real 0.968s user 0.001s sys 0.012s
W20260619 18:47:52.144109 11735 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:47:51.170162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42476 (local address 127.3.243.126:41539)
0619 18:47:51.170366 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:51.170372 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:51.170573 (+ 201us) server_negotiation.cc:407] Connection header received
0619 18:47:51.170728 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:51.170731 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:51.170805 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:51.170891 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:51.172108 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:51.172839 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:51.174021 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:51.174441 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:51.174529 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:52.143853 (+969324us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42476: BlockingRecv error: recv got EOF from 127.0.0.1:42476 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Jun 19 18:47:52 dist-test-slave-m9wg krb5kdc[11595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894871, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM
I20260619 18:47:52.162961 11735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:52.149901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42488 (local address 127.3.243.126:41539)
0619 18:47:52.150089 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:52.150095 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:52.150399 (+ 304us) server_negotiation.cc:407] Connection header received
0619 18:47:52.150535 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:52.150539 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:52.150631 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:52.150732 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:52.151884 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:52.152653 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:52.153883 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:52.154308 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:52.154401 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:52.158099 (+ 3698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:52.158104 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:52.158108 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:52.158141 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:52.160662 (+ 2521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:52.161315 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:52.161319 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:52.161322 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:52.161442 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:52.161764 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:52.161767 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:52.161769 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:52.162018 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:52.162232 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:52.162396 (+ 164us) server_negotiation.cc:299] Negotiation successful
0619 18:47:52.162647 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 18:47:52.166843 11640 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42488:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:47:52.173573 11640 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:47:52.196209 12147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:52.185843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33140 (local address 127.3.243.67:40189)
0619 18:47:52.188250 (+ 2407us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:52.188259 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:52.188288 (+ 29us) server_negotiation.cc:407] Connection header received
0619 18:47:52.188388 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:52.188395 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:52.188634 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:52.188760 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:47:52.189839 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:52.191098 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:52.193661 (+ 2563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:52.194803 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:52.194884 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:52.195033 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:52.195700 (+ 667us) server_negotiation.cc:299] Negotiation successful
0619 18:47:52.195863 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2186,"thread_start_us":2053,"threads_started":1}
I20260619 18:47:52.196307 12145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:52.185323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.66:45059 (local address 127.0.0.1:38872)
0619 18:47:52.186155 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:52.186202 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:52.186372 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:52.188945 (+ 2573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:52.188950 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:52.188973 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:47:52.189700 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:52.189714 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:52.191405 (+ 1691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:52.191414 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:52.193475 (+ 2061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:52.193490 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:52.195261 (+ 1771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:52.195268 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:52.195491 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:52.195529 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 18:47:52.195638 (+ 109us) client_negotiation.cc:240] Negotiation successful
0619 18:47:52.195848 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"mutex_wait_us":139,"thread_start_us":2288,"threads_started":1}
I20260619 18:47:52.196594 12144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:52.185519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.67:40189 (local address 127.0.0.1:33140)
0619 18:47:52.187885 (+ 2366us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:52.187903 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:52.188015 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:52.188951 (+ 936us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:52.188957 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:52.189005 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:47:52.189591 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:52.189602 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:52.191238 (+ 1636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:52.191242 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:52.193494 (+ 2252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:52.193509 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:52.195230 (+ 1721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:52.195239 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:52.195445 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:52.195528 (+ 83us) client_negotiation.cc:769] Sending connection context
0619 18:47:52.195641 (+ 113us) client_negotiation.cc:240] Negotiation successful
0619 18:47:52.195848 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2231,"mutex_wait_us":130,"spinlock_wait_cycles":1408,"thread_start_us":246,"threads_started":1}
I20260619 18:47:52.197057 12146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:52.185550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38872 (local address 127.3.243.66:45059)
0619 18:47:52.187195 (+ 1645us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:52.187203 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:52.187230 (+ 27us) server_negotiation.cc:407] Connection header received
0619 18:47:52.187330 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:52.188124 (+ 794us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:52.188420 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:52.188627 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:47:52.189880 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:52.191297 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:52.193654 (+ 2357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:52.195035 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:52.195171 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:52.195364 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:52.196641 (+ 1277us) server_negotiation.cc:299] Negotiation successful
0619 18:47:52.196801 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1489,"thread_start_us":1353,"threads_started":1}
I20260619 18:47:52.197569 12148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:52.186978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.65:35771 (local address 127.0.0.1:39282)
0619 18:47:52.189651 (+ 2673us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:52.189668 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:52.189787 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:52.190791 (+ 1004us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:52.190797 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:52.190822 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:47:52.191130 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:52.191139 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:52.192878 (+ 1739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:52.192884 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:52.194935 (+ 2051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:52.194950 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:52.196892 (+ 1942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:52.196899 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:52.197067 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:52.197089 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 18:47:52.197166 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 18:47:52.197277 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2528,"thread_start_us":2307,"threads_started":1}
I20260619 18:47:52.198210 12149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:52.187420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39282 (local address 127.3.243.65:35771)
0619 18:47:52.188550 (+ 1130us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:52.188557 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:52.189778 (+ 1221us) server_negotiation.cc:407] Connection header received
0619 18:47:52.190167 (+ 389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:52.190174 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:52.190369 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:52.190534 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:47:52.191301 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:52.192754 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:52.195092 (+ 2338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:52.196449 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:52.196572 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:52.196756 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:52.197771 (+ 1015us) server_negotiation.cc:299] Negotiation successful
0619 18:47:52.197942 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":986,"thread_start_us":851,"threads_started":1}
I20260619 18:47:52.200460 12055 tablet_service.cc:1467] Processing CreateTablet for tablet 3f03242b01b04bf8aa879f22eb962190 (DEFAULT_TABLE table=test-table [id=8b6d3b9955394ce781f954d5b7a5ed0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:47:52.200460 11916 tablet_service.cc:1467] Processing CreateTablet for tablet 3f03242b01b04bf8aa879f22eb962190 (DEFAULT_TABLE table=test-table [id=8b6d3b9955394ce781f954d5b7a5ed0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:47:52.200966 11777 tablet_service.cc:1467] Processing CreateTablet for tablet 3f03242b01b04bf8aa879f22eb962190 (DEFAULT_TABLE table=test-table [id=8b6d3b9955394ce781f954d5b7a5ed0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:47:52.201923 11916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f03242b01b04bf8aa879f22eb962190. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:47:52.201933 12055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f03242b01b04bf8aa879f22eb962190. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:47:52.202071 11777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f03242b01b04bf8aa879f22eb962190. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:47:52.212460 12150 tablet_bootstrap.cc:492] T 3f03242b01b04bf8aa879f22eb962190 P 653a69448f244a2a99faad646667dda7: Bootstrap starting.
I20260619 18:47:52.215782 12152 tablet_bootstrap.cc:492] T 3f03242b01b04bf8aa879f22eb962190 P 9ceae0288d68442b9b808ed536256768: Bootstrap starting.
I20260619 18:47:52.216545 12151 tablet_bootstrap.cc:492] T 3f03242b01b04bf8aa879f22eb962190 P a236998d0ba646d693900cf720e416a1: Bootstrap starting.
I20260619 18:47:52.217214 12150 tablet_bootstrap.cc:654] T 3f03242b01b04bf8aa879f22eb962190 P 653a69448f244a2a99faad646667dda7: Neither blocks nor log segments found. Creating new log.
I20260619 18:47:52.218464 12150 log.cc:824] T 3f03242b01b04bf8aa879f22eb962190 P 653a69448f244a2a99faad646667dda7: Log is configured to *not* fsync() on all Append() calls
I20260619 18:47:52.219818 12151 tablet_bootstrap.cc:654] T 3f03242b01b04bf8aa879f22eb962190 P a236998d0ba646d693900cf720e416a1: Neither blocks nor log segments found. Creating new log.
I20260619 18:47:52.220978 12150 tablet_bootstrap.cc:492] T 3f03242b01b04bf8aa879f22eb962190 P 653a69448f244a2a99faad646667dda7: No bootstrap required, opened a new log
I20260619 18:47:52.221074 12151 log.cc:824] T 3f03242b01b04bf8aa879f22eb962190 P a236998d0ba646d693900cf720e416a1: Log is configured to *not* fsync() on all Append() calls
I20260619 18:47:52.221225 12150 ts_tablet_manager.cc:1397] T 3f03242b01b04bf8aa879f22eb962190 P 653a69448f244a2a99faad646667dda7: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260619 18:47:52.222119 12152 tablet_bootstrap.cc:654] T 3f03242b01b04bf8aa879f22eb962190 P 9ceae0288d68442b9b808ed536256768: Neither blocks nor log segments found. Creating new log.
I20260619 18:47:52.223220 12152 log.cc:824] T 3f03242b01b04bf8aa879f22eb962190 P 9ceae0288d68442b9b808ed536256768: Log is configured to *not* fsync() on all Append() calls
I20260619 18:47:52.223948 12151 tablet_bootstrap.cc:492] T 3f03242b01b04bf8aa879f22eb962190 P a236998d0ba646d693900cf720e416a1: No bootstrap required, opened a new log
I20260619 18:47:52.224243 12151 ts_tablet_manager.cc:1397] T 3f03242b01b04bf8aa879f22eb962190 P a236998d0ba646d693900cf720e416a1: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.003s
I20260619 18:47:52.226946 12152 tablet_bootstrap.cc:492] T 3f03242b01b04bf8aa879f22eb962190 P 9ceae0288d68442b9b808ed536256768: No bootstrap required, opened a new log
I20260619 18:47:52.227193 12152 ts_tablet_manager.cc:1397] T 3f03242b01b04bf8aa879f22eb962190 P 9ceae0288d68442b9b808ed536256768: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.000s
I20260619 18:47:52.230268 12150 raft_consensus.cc:348] T 3f03242b01b04bf8aa879f22eb962190 P 653a69448f244a2a99faad646667dda7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ceae0288d68442b9b808ed536256768" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 45059 } } peers { permanent_uuid: "a236998d0ba646d693900cf720e416a1" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40189 } } peers { permanent_uuid: "653a69448f244a2a99faad646667dda7" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 35771 } }
I20260619 18:47:52.230682 12150 raft_consensus.cc:374] T 3f03242b01b04bf8aa879f22eb962190 P 653a69448f244a2a99faad646667dda7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:47:52.230825 12150 raft_consensus.cc:729] T 3f03242b01b04bf8aa879f22eb962190 P 653a69448f244a2a99faad646667dda7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 653a69448f244a2a99faad646667dda7, State: Initialized, Role: FOLLOWER
I20260619 18:47:52.231158 12151 raft_consensus.cc:348] T 3f03242b01b04bf8aa879f22eb962190 P a236998d0ba646d693900cf720e416a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ceae0288d68442b9b808ed536256768" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 45059 } } peers { permanent_uuid: "a236998d0ba646d693900cf720e416a1" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40189 } } peers { permanent_uuid: "653a69448f244a2a99faad646667dda7" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 35771 } }
I20260619 18:47:52.231477 12151 raft_consensus.cc:374] T 3f03242b01b04bf8aa879f22eb962190 P a236998d0ba646d693900cf720e416a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:47:52.231570 12151 raft_consensus.cc:729] T 3f03242b01b04bf8aa879f22eb962190 P a236998d0ba646d693900cf720e416a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a236998d0ba646d693900cf720e416a1, State: Initialized, Role: FOLLOWER
I20260619 18:47:52.231408 12150 consensus_queue.cc:260] T 3f03242b01b04bf8aa879f22eb962190 P 653a69448f244a2a99faad646667dda7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ceae0288d68442b9b808ed536256768" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 45059 } } peers { permanent_uuid: "a236998d0ba646d693900cf720e416a1" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40189 } } peers { permanent_uuid: "653a69448f244a2a99faad646667dda7" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 35771 } }
I20260619 18:47:52.232110 12151 consensus_queue.cc:260] T 3f03242b01b04bf8aa879f22eb962190 P a236998d0ba646d693900cf720e416a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ceae0288d68442b9b808ed536256768" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 45059 } } peers { permanent_uuid: "a236998d0ba646d693900cf720e416a1" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40189 } } peers { permanent_uuid: "653a69448f244a2a99faad646667dda7" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 35771 } }
I20260619 18:47:52.233093 12150 ts_tablet_manager.cc:1428] T 3f03242b01b04bf8aa879f22eb962190 P 653a69448f244a2a99faad646667dda7: Time spent starting tablet: real 0.012s user 0.008s sys 0.005s
I20260619 18:47:52.234480 12151 ts_tablet_manager.cc:1428] T 3f03242b01b04bf8aa879f22eb962190 P a236998d0ba646d693900cf720e416a1: Time spent starting tablet: real 0.010s user 0.008s sys 0.001s
I20260619 18:47:52.234942 12157 raft_consensus.cc:482] T 3f03242b01b04bf8aa879f22eb962190 P a236998d0ba646d693900cf720e416a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:47:52.235179 12157 raft_consensus.cc:504] T 3f03242b01b04bf8aa879f22eb962190 P a236998d0ba646d693900cf720e416a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ceae0288d68442b9b808ed536256768" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 45059 } } peers { permanent_uuid: "a236998d0ba646d693900cf720e416a1" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40189 } } peers { permanent_uuid: "653a69448f244a2a99faad646667dda7" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 35771 } }
I20260619 18:47:52.236982 12152 raft_consensus.cc:348] T 3f03242b01b04bf8aa879f22eb962190 P 9ceae0288d68442b9b808ed536256768 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ceae0288d68442b9b808ed536256768" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 45059 } } peers { permanent_uuid: "a236998d0ba646d693900cf720e416a1" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40189 } } peers { permanent_uuid: "653a69448f244a2a99faad646667dda7" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 35771 } }
I20260619 18:47:52.237461 12152 raft_consensus.cc:374] T 3f03242b01b04bf8aa879f22eb962190 P 9ceae0288d68442b9b808ed536256768 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:47:52.237603 12152 raft_consensus.cc:729] T 3f03242b01b04bf8aa879f22eb962190 P 9ceae0288d68442b9b808ed536256768 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ceae0288d68442b9b808ed536256768, State: Initialized, Role: FOLLOWER
I20260619 18:47:52.238317 12152 consensus_queue.cc:260] T 3f03242b01b04bf8aa879f22eb962190 P 9ceae0288d68442b9b808ed536256768 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ceae0288d68442b9b808ed536256768" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 45059 } } peers { permanent_uuid: "a236998d0ba646d693900cf720e416a1" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40189 } } peers { permanent_uuid: "653a69448f244a2a99faad646667dda7" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 35771 } }
I20260619 18:47:52.241143 12157 leader_election.cc:288] T 3f03242b01b04bf8aa879f22eb962190 P a236998d0ba646d693900cf720e416a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ceae0288d68442b9b808ed536256768 (127.3.243.66:45059), 653a69448f244a2a99faad646667dda7 (127.3.243.65:35771)
I20260619 18:47:52.242871 12152 ts_tablet_manager.cc:1428] T 3f03242b01b04bf8aa879f22eb962190 P 9ceae0288d68442b9b808ed536256768: Time spent starting tablet: real 0.016s user 0.013s sys 0.003s
I20260619 18:47:52.250121 12146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:52.242421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.67:56485 (local address 127.3.243.66:45059)
0619 18:47:52.242614 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:52.242619 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:52.243674 (+ 1055us) server_negotiation.cc:407] Connection header received
0619 18:47:52.243927 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:52.243932 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:52.244019 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:52.244074 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:47:52.244896 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:52.245891 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:52.248497 (+ 2606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:52.249296 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:52.249409 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:52.249475 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:52.249755 (+ 280us) server_negotiation.cc:299] Negotiation successful
0619 18:47:52.249873 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 18:47:52.250185 12161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:52.242218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.66:45059 (local address 127.3.243.67:56485)
0619 18:47:52.243578 (+ 1360us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:52.243606 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:52.243768 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:52.244272 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:52.244277 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:52.244301 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:47:52.244627 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:52.244680 (+ 53us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:52.246026 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:52.246031 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:52.248351 (+ 2320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:52.248359 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:52.249452 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:52.249456 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:52.249605 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:52.249632 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 18:47:52.249706 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 18:47:52.249827 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"mutex_wait_us":82,"thread_start_us":534,"threads_started":1}
I20260619 18:47:52.251303 12149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:52.237374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.67:36373 (local address 127.3.243.65:35771)
0619 18:47:52.237566 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:52.237572 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:52.241308 (+ 3736us) server_negotiation.cc:407] Connection header received
0619 18:47:52.243489 (+ 2181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:52.243494 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:52.243565 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:52.243650 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:47:52.245233 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:52.246168 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:52.249543 (+ 3375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:52.250473 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:52.250608 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:52.250677 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:52.251048 (+ 371us) server_negotiation.cc:299] Negotiation successful
0619 18:47:52.251150 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 18:47:52.251134 11936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f03242b01b04bf8aa879f22eb962190" candidate_uuid: "a236998d0ba646d693900cf720e416a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ceae0288d68442b9b808ed536256768" is_pre_election: true
I20260619 18:47:52.251691 12159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:52.237311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.65:35771 (local address 127.3.243.67:36373)
0619 18:47:52.241198 (+ 3887us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:52.241218 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:52.243329 (+ 2111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:52.243807 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:52.243812 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:52.243838 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:47:52.245103 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:52.245114 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:52.246289 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:52.246295 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:52.248344 (+ 2049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:52.248359 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:52.250689 (+ 2330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:52.250703 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:52.250867 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:52.250898 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 18:47:52.250996 (+ 98us) client_negotiation.cc:240] Negotiation successful
0619 18:47:52.251115 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3726,"mutex_wait_us":23,"thread_start_us":3563,"threads_started":1}
I20260619 18:47:52.251637 11936 raft_consensus.cc:2454] T 3f03242b01b04bf8aa879f22eb962190 P 9ceae0288d68442b9b808ed536256768 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a236998d0ba646d693900cf720e416a1 in term 0.
I20260619 18:47:52.252337 12009 leader_election.cc:302] T 3f03242b01b04bf8aa879f22eb962190 P a236998d0ba646d693900cf720e416a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ceae0288d68442b9b808ed536256768, a236998d0ba646d693900cf720e416a1; no voters:
I20260619 18:47:52.252733 12158 raft_consensus.cc:2781] T 3f03242b01b04bf8aa879f22eb962190 P a236998d0ba646d693900cf720e416a1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:47:52.252676 11797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f03242b01b04bf8aa879f22eb962190" candidate_uuid: "a236998d0ba646d693900cf720e416a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "653a69448f244a2a99faad646667dda7" is_pre_election: true
I20260619 18:47:52.252851 12158 raft_consensus.cc:482] T 3f03242b01b04bf8aa879f22eb962190 P a236998d0ba646d693900cf720e416a1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:47:52.252957 12158 raft_consensus.cc:3037] T 3f03242b01b04bf8aa879f22eb962190 P a236998d0ba646d693900cf720e416a1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:47:52.253111 11797 raft_consensus.cc:2454] T 3f03242b01b04bf8aa879f22eb962190 P 653a69448f244a2a99faad646667dda7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a236998d0ba646d693900cf720e416a1 in term 0.
I20260619 18:47:52.255242 12158 raft_consensus.cc:504] T 3f03242b01b04bf8aa879f22eb962190 P a236998d0ba646d693900cf720e416a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ceae0288d68442b9b808ed536256768" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 45059 } } peers { permanent_uuid: "a236998d0ba646d693900cf720e416a1" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40189 } } peers { permanent_uuid: "653a69448f244a2a99faad646667dda7" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 35771 } }
I20260619 18:47:52.255852 12158 leader_election.cc:288] T 3f03242b01b04bf8aa879f22eb962190 P a236998d0ba646d693900cf720e416a1 [CANDIDATE]: Term 1 election: Requested vote from peers 9ceae0288d68442b9b808ed536256768 (127.3.243.66:45059), 653a69448f244a2a99faad646667dda7 (127.3.243.65:35771)
I20260619 18:47:52.256215 11936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f03242b01b04bf8aa879f22eb962190" candidate_uuid: "a236998d0ba646d693900cf720e416a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ceae0288d68442b9b808ed536256768"
I20260619 18:47:52.256337 11797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f03242b01b04bf8aa879f22eb962190" candidate_uuid: "a236998d0ba646d693900cf720e416a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "653a69448f244a2a99faad646667dda7"
I20260619 18:47:52.256407 11936 raft_consensus.cc:3037] T 3f03242b01b04bf8aa879f22eb962190 P 9ceae0288d68442b9b808ed536256768 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:47:52.256512 11797 raft_consensus.cc:3037] T 3f03242b01b04bf8aa879f22eb962190 P 653a69448f244a2a99faad646667dda7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:47:52.259392 11936 raft_consensus.cc:2454] T 3f03242b01b04bf8aa879f22eb962190 P 9ceae0288d68442b9b808ed536256768 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a236998d0ba646d693900cf720e416a1 in term 1.
I20260619 18:47:52.259439 11797 raft_consensus.cc:2454] T 3f03242b01b04bf8aa879f22eb962190 P 653a69448f244a2a99faad646667dda7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a236998d0ba646d693900cf720e416a1 in term 1.
I20260619 18:47:52.259783 12009 leader_election.cc:302] T 3f03242b01b04bf8aa879f22eb962190 P a236998d0ba646d693900cf720e416a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ceae0288d68442b9b808ed536256768, a236998d0ba646d693900cf720e416a1; no voters:
I20260619 18:47:52.260038 12158 raft_consensus.cc:2781] T 3f03242b01b04bf8aa879f22eb962190 P a236998d0ba646d693900cf720e416a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:47:52.260224 12158 raft_consensus.cc:686] T 3f03242b01b04bf8aa879f22eb962190 P a236998d0ba646d693900cf720e416a1 [term 1 LEADER]: Becoming Leader. State: Replica: a236998d0ba646d693900cf720e416a1, State: Running, Role: LEADER
I20260619 18:47:52.260648 12158 consensus_queue.cc:237] T 3f03242b01b04bf8aa879f22eb962190 P a236998d0ba646d693900cf720e416a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ceae0288d68442b9b808ed536256768" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 45059 } } peers { permanent_uuid: "a236998d0ba646d693900cf720e416a1" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40189 } } peers { permanent_uuid: "653a69448f244a2a99faad646667dda7" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 35771 } }
I20260619 18:47:52.264995 11639 catalog_manager.cc:5495] T 3f03242b01b04bf8aa879f22eb962190 P a236998d0ba646d693900cf720e416a1 reported cstate change: term changed from 0 to 1, leader changed from <none> to a236998d0ba646d693900cf720e416a1 (127.3.243.67). New cstate: current_term: 1 leader_uuid: "a236998d0ba646d693900cf720e416a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ceae0288d68442b9b808ed536256768" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 45059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a236998d0ba646d693900cf720e416a1" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "653a69448f244a2a99faad646667dda7" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 35771 } health_report { overall_health: UNKNOWN } } }
I20260619 18:47:52.280189 12147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:52.275088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33154 (local address 127.3.243.67:40189)
0619 18:47:52.275282 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:52.275289 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:52.275419 (+ 130us) server_negotiation.cc:407] Connection header received
0619 18:47:52.275589 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:52.275607 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:52.275682 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:52.275807 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:47:52.276213 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:52.276933 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:52.278202 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:52.278602 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:52.278707 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:52.279099 (+ 392us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:47:52.279623 (+ 524us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:47:52.279755 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:52.279841 (+ 86us) server_negotiation.cc:299] Negotiation successful
0619 18:47:52.279918 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 18:47:52.287595 11797 raft_consensus.cc:1261] T 3f03242b01b04bf8aa879f22eb962190 P 653a69448f244a2a99faad646667dda7 [term 1 FOLLOWER]: Refusing update from remote peer a236998d0ba646d693900cf720e416a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:47:52.287633 11936 raft_consensus.cc:1261] T 3f03242b01b04bf8aa879f22eb962190 P 9ceae0288d68442b9b808ed536256768 [term 1 FOLLOWER]: Refusing update from remote peer a236998d0ba646d693900cf720e416a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:47:52.288480 12157 consensus_queue.cc:1035] T 3f03242b01b04bf8aa879f22eb962190 P a236998d0ba646d693900cf720e416a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "653a69448f244a2a99faad646667dda7" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 35771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:47:52.288728 12158 consensus_queue.cc:1035] T 3f03242b01b04bf8aa879f22eb962190 P a236998d0ba646d693900cf720e416a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ceae0288d68442b9b808ed536256768" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 45059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:47:52.318230 11639 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42488:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:47:52.318517 11639 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42488:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:47:52.322078 11639 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 022c3c6be95b46f8bf8e526a33780f6d: Sending DeleteTablet for 3 replicas of tablet 3f03242b01b04bf8aa879f22eb962190
I20260619 18:47:52.323093 12055 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f03242b01b04bf8aa879f22eb962190 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:47:52 UTC) from {username='oryx', principal='oryx/127.3.243.126@KRBTEST.COM'} at 127.0.0.1:33140
I20260619 18:47:52.323153 11916 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f03242b01b04bf8aa879f22eb962190 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:47:52 UTC) from {username='oryx', principal='oryx/127.3.243.126@KRBTEST.COM'} at 127.0.0.1:38872
I20260619 18:47:52.324000 11639 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42488:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 18:47:52.324301 11777 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f03242b01b04bf8aa879f22eb962190 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:47:52 UTC) from {username='oryx', principal='oryx/127.3.243.126@KRBTEST.COM'} at 127.0.0.1:39282
W20260619 18:47:52.324589 11639 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:47:52.323706 12054 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f03242b01b04bf8aa879f22eb962190 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:47:52 UTC) from {username='oryx', principal='oryx/127.3.243.126@KRBTEST.COM'} at 127.0.0.1:33140
I20260619 18:47:52.331044 11776 tablet_service.cc:1467] Processing CreateTablet for tablet c0e9d3eca12e49b4967b368bbc068b85 (DEFAULT_TABLE table=test-table [id=8e87f934554744baa7bb092e2bf348b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:47:52.331346 11915 tablet_service.cc:1467] Processing CreateTablet for tablet c0e9d3eca12e49b4967b368bbc068b85 (DEFAULT_TABLE table=test-table [id=8e87f934554744baa7bb092e2bf348b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:47:52.331571 11776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0e9d3eca12e49b4967b368bbc068b85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:47:52.331820 11915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0e9d3eca12e49b4967b368bbc068b85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:47:52.336781 12054 tablet_service.cc:1467] Processing CreateTablet for tablet c0e9d3eca12e49b4967b368bbc068b85 (DEFAULT_TABLE table=test-table [id=8e87f934554744baa7bb092e2bf348b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:47:52.337311 12054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0e9d3eca12e49b4967b368bbc068b85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:47:52.337678 12152 tablet_bootstrap.cc:492] T c0e9d3eca12e49b4967b368bbc068b85 P 9ceae0288d68442b9b808ed536256768: Bootstrap starting.
I20260619 18:47:52.339177 12152 tablet_bootstrap.cc:654] T c0e9d3eca12e49b4967b368bbc068b85 P 9ceae0288d68442b9b808ed536256768: Neither blocks nor log segments found. Creating new log.
I20260619 18:47:52.346628 12151 tablet_bootstrap.cc:492] T c0e9d3eca12e49b4967b368bbc068b85 P a236998d0ba646d693900cf720e416a1: Bootstrap starting.
I20260619 18:47:52.348416 12151 tablet_bootstrap.cc:654] T c0e9d3eca12e49b4967b368bbc068b85 P a236998d0ba646d693900cf720e416a1: Neither blocks nor log segments found. Creating new log.
I20260619 18:47:52.351647 12150 tablet_bootstrap.cc:492] T c0e9d3eca12e49b4967b368bbc068b85 P 653a69448f244a2a99faad646667dda7: Bootstrap starting.
I20260619 18:47:52.353287 12150 tablet_bootstrap.cc:654] T c0e9d3eca12e49b4967b368bbc068b85 P 653a69448f244a2a99faad646667dda7: Neither blocks nor log segments found. Creating new log.
I20260619 18:47:52.369028 12151 tablet_bootstrap.cc:492] T c0e9d3eca12e49b4967b368bbc068b85 P a236998d0ba646d693900cf720e416a1: No bootstrap required, opened a new log
I20260619 18:47:52.369333 12151 ts_tablet_manager.cc:1397] T c0e9d3eca12e49b4967b368bbc068b85 P a236998d0ba646d693900cf720e416a1: Time spent bootstrapping tablet: real 0.023s user 0.007s sys 0.005s
I20260619 18:47:52.370193 12151 raft_consensus.cc:348] T c0e9d3eca12e49b4967b368bbc068b85 P a236998d0ba646d693900cf720e416a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653a69448f244a2a99faad646667dda7" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 35771 } } peers { permanent_uuid: "9ceae0288d68442b9b808ed536256768" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 45059 } } peers { permanent_uuid: "a236998d0ba646d693900cf720e416a1" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40189 } }
I20260619 18:47:52.370497 12151 raft_consensus.cc:374] T c0e9d3eca12e49b4967b368bbc068b85 P a236998d0ba646d693900cf720e416a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:47:52.370657 12151 raft_consensus.cc:729] T c0e9d3eca12e49b4967b368bbc068b85 P a236998d0ba646d693900cf720e416a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a236998d0ba646d693900cf720e416a1, State: Initialized, Role: FOLLOWER
I20260619 18:47:52.371035 12151 consensus_queue.cc:260] T c0e9d3eca12e49b4967b368bbc068b85 P a236998d0ba646d693900cf720e416a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653a69448f244a2a99faad646667dda7" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 35771 } } peers { permanent_uuid: "9ceae0288d68442b9b808ed536256768" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 45059 } } peers { permanent_uuid: "a236998d0ba646d693900cf720e416a1" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40189 } }
I20260619 18:47:52.371682 12151 ts_tablet_manager.cc:1428] T c0e9d3eca12e49b4967b368bbc068b85 P a236998d0ba646d693900cf720e416a1: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20260619 18:47:52.374897 12126 tablet.cc:2369] T c0e9d3eca12e49b4967b368bbc068b85 P a236998d0ba646d693900cf720e416a1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:47:52.376047 12176 tablet_replica.cc:330] T 3f03242b01b04bf8aa879f22eb962190 P 653a69448f244a2a99faad646667dda7: stopping tablet replica
I20260619 18:47:52.376544 12176 raft_consensus.cc:2229] T 3f03242b01b04bf8aa879f22eb962190 P 653a69448f244a2a99faad646667dda7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:47:52.377071 12176 raft_consensus.cc:2258] T 3f03242b01b04bf8aa879f22eb962190 P 653a69448f244a2a99faad646667dda7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:47:52.377988 12175 tablet_replica.cc:330] T 3f03242b01b04bf8aa879f22eb962190 P a236998d0ba646d693900cf720e416a1: stopping tablet replica
I20260619 18:47:52.378350 12175 raft_consensus.cc:2229] T 3f03242b01b04bf8aa879f22eb962190 P a236998d0ba646d693900cf720e416a1 [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:47:52.378631 12150 tablet_bootstrap.cc:492] T c0e9d3eca12e49b4967b368bbc068b85 P 653a69448f244a2a99faad646667dda7: No bootstrap required, opened a new log
I20260619 18:47:52.378773 12150 ts_tablet_manager.cc:1397] T c0e9d3eca12e49b4967b368bbc068b85 P 653a69448f244a2a99faad646667dda7: Time spent bootstrapping tablet: real 0.027s user 0.011s sys 0.001s
I20260619 18:47:52.379009 12175 raft_consensus.cc:2258] T 3f03242b01b04bf8aa879f22eb962190 P a236998d0ba646d693900cf720e416a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:47:52.379761 12150 raft_consensus.cc:348] T c0e9d3eca12e49b4967b368bbc068b85 P 653a69448f244a2a99faad646667dda7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653a69448f244a2a99faad646667dda7" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 35771 } } peers { permanent_uuid: "9ceae0288d68442b9b808ed536256768" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 45059 } } peers { permanent_uuid: "a236998d0ba646d693900cf720e416a1" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40189 } }
I20260619 18:47:52.379976 12150 raft_consensus.cc:374] T c0e9d3eca12e49b4967b368bbc068b85 P 653a69448f244a2a99faad646667dda7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:47:52.380070 12150 raft_consensus.cc:729] T c0e9d3eca12e49b4967b368bbc068b85 P 653a69448f244a2a99faad646667dda7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 653a69448f244a2a99faad646667dda7, State: Initialized, Role: FOLLOWER
I20260619 18:47:52.380476 12150 consensus_queue.cc:260] T c0e9d3eca12e49b4967b368bbc068b85 P 653a69448f244a2a99faad646667dda7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653a69448f244a2a99faad646667dda7" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 35771 } } peers { permanent_uuid: "9ceae0288d68442b9b808ed536256768" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 45059 } } peers { permanent_uuid: "a236998d0ba646d693900cf720e416a1" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40189 } }
I20260619 18:47:52.381000 12150 ts_tablet_manager.cc:1428] T c0e9d3eca12e49b4967b368bbc068b85 P 653a69448f244a2a99faad646667dda7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 18:47:52.386319 12175 ts_tablet_manager.cc:1905] T 3f03242b01b04bf8aa879f22eb962190 P a236998d0ba646d693900cf720e416a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:47:52.389308 12152 tablet_bootstrap.cc:492] T c0e9d3eca12e49b4967b368bbc068b85 P 9ceae0288d68442b9b808ed536256768: No bootstrap required, opened a new log
I20260619 18:47:52.389462 12152 ts_tablet_manager.cc:1397] T c0e9d3eca12e49b4967b368bbc068b85 P 9ceae0288d68442b9b808ed536256768: Time spent bootstrapping tablet: real 0.052s user 0.016s sys 0.030s
I20260619 18:47:52.389504 12176 ts_tablet_manager.cc:1905] T 3f03242b01b04bf8aa879f22eb962190 P 653a69448f244a2a99faad646667dda7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:47:52.390362 12152 raft_consensus.cc:348] T c0e9d3eca12e49b4967b368bbc068b85 P 9ceae0288d68442b9b808ed536256768 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653a69448f244a2a99faad646667dda7" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 35771 } } peers { permanent_uuid: "9ceae0288d68442b9b808ed536256768" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 45059 } } peers { permanent_uuid: "a236998d0ba646d693900cf720e416a1" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40189 } }
I20260619 18:47:52.390583 12152 raft_consensus.cc:374] T c0e9d3eca12e49b4967b368bbc068b85 P 9ceae0288d68442b9b808ed536256768 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:47:52.390686 12152 raft_consensus.cc:729] T c0e9d3eca12e49b4967b368bbc068b85 P 9ceae0288d68442b9b808ed536256768 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ceae0288d68442b9b808ed536256768, State: Initialized, Role: FOLLOWER
W20260619 18:47:52.391114 11848 tablet.cc:2369] T c0e9d3eca12e49b4967b368bbc068b85 P 653a69448f244a2a99faad646667dda7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:47:52.391841 12152 consensus_queue.cc:260] T c0e9d3eca12e49b4967b368bbc068b85 P 9ceae0288d68442b9b808ed536256768 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653a69448f244a2a99faad646667dda7" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 35771 } } peers { permanent_uuid: "9ceae0288d68442b9b808ed536256768" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 45059 } } peers { permanent_uuid: "a236998d0ba646d693900cf720e416a1" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40189 } }
I20260619 18:47:52.392628 12152 ts_tablet_manager.cc:1428] T c0e9d3eca12e49b4967b368bbc068b85 P 9ceae0288d68442b9b808ed536256768: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 18:47:52.395360 12176 ts_tablet_manager.cc:1918] T 3f03242b01b04bf8aa879f22eb962190 P 653a69448f244a2a99faad646667dda7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:47:52.395491 12176 log.cc:1196] T 3f03242b01b04bf8aa879f22eb962190 P 653a69448f244a2a99faad646667dda7: Deleting WAL directory at /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/minicluster-data/ts-0/wal/wals/3f03242b01b04bf8aa879f22eb962190
I20260619 18:47:52.395895 12176 ts_tablet_manager.cc:1939] T 3f03242b01b04bf8aa879f22eb962190 P 653a69448f244a2a99faad646667dda7: Deleting consensus metadata
I20260619 18:47:52.400166 12175 ts_tablet_manager.cc:1918] T 3f03242b01b04bf8aa879f22eb962190 P a236998d0ba646d693900cf720e416a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:47:52.400328 12175 log.cc:1196] T 3f03242b01b04bf8aa879f22eb962190 P a236998d0ba646d693900cf720e416a1: Deleting WAL directory at /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/minicluster-data/ts-2/wal/wals/3f03242b01b04bf8aa879f22eb962190
I20260619 18:47:52.400869 12175 ts_tablet_manager.cc:1939] T 3f03242b01b04bf8aa879f22eb962190 P a236998d0ba646d693900cf720e416a1: Deleting consensus metadata
I20260619 18:47:52.400887 12174 tablet_replica.cc:330] T 3f03242b01b04bf8aa879f22eb962190 P 9ceae0288d68442b9b808ed536256768: stopping tablet replica
I20260619 18:47:52.401402 12174 raft_consensus.cc:2229] T 3f03242b01b04bf8aa879f22eb962190 P 9ceae0288d68442b9b808ed536256768 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:47:52.401974 12174 raft_consensus.cc:2258] T 3f03242b01b04bf8aa879f22eb962190 P 9ceae0288d68442b9b808ed536256768 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:47:52.402179 11625 catalog_manager.cc:4841] TS 653a69448f244a2a99faad646667dda7 (127.3.243.65:35771): tablet 3f03242b01b04bf8aa879f22eb962190 (table test-table [id=8b6d3b9955394ce781f954d5b7a5ed0e]) successfully deleted
I20260619 18:47:52.402483 11625 catalog_manager.cc:4841] TS a236998d0ba646d693900cf720e416a1 (127.3.243.67:40189): tablet 3f03242b01b04bf8aa879f22eb962190 (table test-table [id=8b6d3b9955394ce781f954d5b7a5ed0e]) successfully deleted
W20260619 18:47:52.403445 11625 catalog_manager.cc:4808] TS a236998d0ba646d693900cf720e416a1 (127.3.243.67:40189): delete failed for tablet 3f03242b01b04bf8aa879f22eb962190 because the tablet was not found. No further retry: Not found: Tablet not found: 3f03242b01b04bf8aa879f22eb962190
I20260619 18:47:52.403981 12174 ts_tablet_manager.cc:1905] T 3f03242b01b04bf8aa879f22eb962190 P 9ceae0288d68442b9b808ed536256768: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:47:52.408277 12174 ts_tablet_manager.cc:1918] T 3f03242b01b04bf8aa879f22eb962190 P 9ceae0288d68442b9b808ed536256768: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:47:52.408397 12174 log.cc:1196] T 3f03242b01b04bf8aa879f22eb962190 P 9ceae0288d68442b9b808ed536256768: Deleting WAL directory at /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/minicluster-data/ts-1/wal/wals/3f03242b01b04bf8aa879f22eb962190
I20260619 18:47:52.408782 12174 ts_tablet_manager.cc:1939] T 3f03242b01b04bf8aa879f22eb962190 P 9ceae0288d68442b9b808ed536256768: Deleting consensus metadata
I20260619 18:47:52.410085 11627 catalog_manager.cc:4841] TS 9ceae0288d68442b9b808ed536256768 (127.3.243.66:45059): tablet 3f03242b01b04bf8aa879f22eb962190 (table test-table [id=8b6d3b9955394ce781f954d5b7a5ed0e]) successfully deleted
I20260619 18:47:52.518060 12156 raft_consensus.cc:482] T c0e9d3eca12e49b4967b368bbc068b85 P 653a69448f244a2a99faad646667dda7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:47:52.518306 12156 raft_consensus.cc:504] T c0e9d3eca12e49b4967b368bbc068b85 P 653a69448f244a2a99faad646667dda7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653a69448f244a2a99faad646667dda7" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 35771 } } peers { permanent_uuid: "9ceae0288d68442b9b808ed536256768" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 45059 } } peers { permanent_uuid: "a236998d0ba646d693900cf720e416a1" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40189 } }
I20260619 18:47:52.520119 12156 leader_election.cc:288] T c0e9d3eca12e49b4967b368bbc068b85 P 653a69448f244a2a99faad646667dda7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ceae0288d68442b9b808ed536256768 (127.3.243.66:45059), a236998d0ba646d693900cf720e416a1 (127.3.243.67:40189)
I20260619 18:47:52.527631 12147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:52.520968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.65:41167 (local address 127.3.243.67:40189)
0619 18:47:52.521139 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:52.521147 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:52.521804 (+ 657us) server_negotiation.cc:407] Connection header received
0619 18:47:52.521945 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:52.521951 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:52.522041 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:52.522138 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:47:52.522819 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:52.523880 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:52.525733 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:52.526665 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:52.526819 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:52.526889 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:52.527196 (+ 307us) server_negotiation.cc:299] Negotiation successful
0619 18:47:52.527323 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 18:47:52.527671 12181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:52.520574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.67:40189 (local address 127.3.243.65:41167)
0619 18:47:52.521722 (+ 1148us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:52.521735 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:52.521829 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:52.522171 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:52.522176 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:52.522205 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:47:52.522664 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:52.522677 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:52.524015 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:52.524019 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:52.525579 (+ 1560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:52.525598 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:52.526868 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:52.526874 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:52.527037 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:52.527071 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 18:47:52.527163 (+ 92us) client_negotiation.cc:240] Negotiation successful
0619 18:47:52.527289 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1031,"mutex_wait_us":32,"thread_start_us":752,"threads_started":1}
I20260619 18:47:52.528649 12146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:52.520750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.65:38609 (local address 127.3.243.66:45059)
0619 18:47:52.520917 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:52.520924 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:52.521461 (+ 537us) server_negotiation.cc:407] Connection header received
0619 18:47:52.521746 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:52.521754 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:52.521841 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:52.521933 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:47:52.522696 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:52.523833 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:52.526357 (+ 2524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:52.527386 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:52.527537 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:52.527613 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:52.528252 (+ 639us) server_negotiation.cc:299] Negotiation successful
0619 18:47:52.528384 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 18:47:52.528679 12180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:52.520346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.66:45059 (local address 127.3.243.65:38609)
0619 18:47:52.521312 (+ 966us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:52.521331 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:52.521588 (+ 257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:52.521950 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:52.521954 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:52.521975 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:47:52.522485 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:52.522497 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:52.523983 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:52.523988 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:52.526183 (+ 2195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:52.526199 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:52.527894 (+ 1695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:52.527900 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:52.528069 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:52.528102 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 18:47:52.528225 (+ 123us) client_negotiation.cc:240] Negotiation successful
0619 18:47:52.528341 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":830,"mutex_wait_us":37,"thread_start_us":665,"threads_started":1}
I20260619 18:47:52.528592 12075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0e9d3eca12e49b4967b368bbc068b85" candidate_uuid: "653a69448f244a2a99faad646667dda7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a236998d0ba646d693900cf720e416a1" is_pre_election: true
I20260619 18:47:52.529177 12075 raft_consensus.cc:2454] T c0e9d3eca12e49b4967b368bbc068b85 P a236998d0ba646d693900cf720e416a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 653a69448f244a2a99faad646667dda7 in term 0.
I20260619 18:47:52.529449 11936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0e9d3eca12e49b4967b368bbc068b85" candidate_uuid: "653a69448f244a2a99faad646667dda7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ceae0288d68442b9b808ed536256768" is_pre_election: true
I20260619 18:47:52.529685 11936 raft_consensus.cc:2454] T c0e9d3eca12e49b4967b368bbc068b85 P 9ceae0288d68442b9b808ed536256768 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 653a69448f244a2a99faad646667dda7 in term 0.
I20260619 18:47:52.529748 11728 leader_election.cc:302] T c0e9d3eca12e49b4967b368bbc068b85 P 653a69448f244a2a99faad646667dda7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 653a69448f244a2a99faad646667dda7, a236998d0ba646d693900cf720e416a1; no voters:
I20260619 18:47:52.530130 12156 raft_consensus.cc:2781] T c0e9d3eca12e49b4967b368bbc068b85 P 653a69448f244a2a99faad646667dda7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:47:52.530246 12156 raft_consensus.cc:482] T c0e9d3eca12e49b4967b368bbc068b85 P 653a69448f244a2a99faad646667dda7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:47:52.530341 12156 raft_consensus.cc:3037] T c0e9d3eca12e49b4967b368bbc068b85 P 653a69448f244a2a99faad646667dda7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:47:52.532380 12156 raft_consensus.cc:504] T c0e9d3eca12e49b4967b368bbc068b85 P 653a69448f244a2a99faad646667dda7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653a69448f244a2a99faad646667dda7" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 35771 } } peers { permanent_uuid: "9ceae0288d68442b9b808ed536256768" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 45059 } } peers { permanent_uuid: "a236998d0ba646d693900cf720e416a1" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40189 } }
I20260619 18:47:52.532955 12156 leader_election.cc:288] T c0e9d3eca12e49b4967b368bbc068b85 P 653a69448f244a2a99faad646667dda7 [CANDIDATE]: Term 1 election: Requested vote from peers 9ceae0288d68442b9b808ed536256768 (127.3.243.66:45059), a236998d0ba646d693900cf720e416a1 (127.3.243.67:40189)
I20260619 18:47:52.533370 11936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0e9d3eca12e49b4967b368bbc068b85" candidate_uuid: "653a69448f244a2a99faad646667dda7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ceae0288d68442b9b808ed536256768"
I20260619 18:47:52.533556 11936 raft_consensus.cc:3037] T c0e9d3eca12e49b4967b368bbc068b85 P 9ceae0288d68442b9b808ed536256768 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:47:52.535437 11936 raft_consensus.cc:2454] T c0e9d3eca12e49b4967b368bbc068b85 P 9ceae0288d68442b9b808ed536256768 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 653a69448f244a2a99faad646667dda7 in term 1.
I20260619 18:47:52.535781 11730 leader_election.cc:302] T c0e9d3eca12e49b4967b368bbc068b85 P 653a69448f244a2a99faad646667dda7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 653a69448f244a2a99faad646667dda7, 9ceae0288d68442b9b808ed536256768; no voters:
I20260619 18:47:52.535740 12075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0e9d3eca12e49b4967b368bbc068b85" candidate_uuid: "653a69448f244a2a99faad646667dda7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a236998d0ba646d693900cf720e416a1"
I20260619 18:47:52.535939 12075 raft_consensus.cc:3037] T c0e9d3eca12e49b4967b368bbc068b85 P a236998d0ba646d693900cf720e416a1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:47:52.536033 12156 raft_consensus.cc:2781] T c0e9d3eca12e49b4967b368bbc068b85 P 653a69448f244a2a99faad646667dda7 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:47:52.536490 12156 raft_consensus.cc:686] T c0e9d3eca12e49b4967b368bbc068b85 P 653a69448f244a2a99faad646667dda7 [term 1 LEADER]: Becoming Leader. State: Replica: 653a69448f244a2a99faad646667dda7, State: Running, Role: LEADER
I20260619 18:47:52.536878 12156 consensus_queue.cc:237] T c0e9d3eca12e49b4967b368bbc068b85 P 653a69448f244a2a99faad646667dda7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653a69448f244a2a99faad646667dda7" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 35771 } } peers { permanent_uuid: "9ceae0288d68442b9b808ed536256768" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 45059 } } peers { permanent_uuid: "a236998d0ba646d693900cf720e416a1" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40189 } }
I20260619 18:47:52.539126 12075 raft_consensus.cc:2454] T c0e9d3eca12e49b4967b368bbc068b85 P a236998d0ba646d693900cf720e416a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 653a69448f244a2a99faad646667dda7 in term 1.
I20260619 18:47:52.540053 11640 catalog_manager.cc:5495] T c0e9d3eca12e49b4967b368bbc068b85 P 653a69448f244a2a99faad646667dda7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 653a69448f244a2a99faad646667dda7 (127.3.243.65). New cstate: current_term: 1 leader_uuid: "653a69448f244a2a99faad646667dda7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653a69448f244a2a99faad646667dda7" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 35771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ceae0288d68442b9b808ed536256768" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 45059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a236998d0ba646d693900cf720e416a1" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40189 } health_report { overall_health: UNKNOWN } } }
Jun 19 18:47:52 dist-test-slave-m9wg krb5kdc[11595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894869, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.243.126@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM
I20260619 18:47:52.621613 11735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:52.606758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42502 (local address 127.3.243.126:41539)
0619 18:47:52.607011 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:52.607018 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:52.607514 (+ 496us) server_negotiation.cc:407] Connection header received
0619 18:47:52.607648 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:52.607654 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:52.607733 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:52.607848 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:52.608994 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:52.610113 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:52.611376 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:52.612026 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:52.612210 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:52.616131 (+ 3921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:52.616136 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:52.616139 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:52.616201 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:52.618597 (+ 2396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:52.619456 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:52.619460 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:52.619462 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:52.619576 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:52.620012 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:52.620016 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:52.620018 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:52.620328 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:52.620518 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:52.621015 (+ 497us) server_negotiation.cc:299] Negotiation successful
0619 18:47:52.621286 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":148,"server-negotiator.queue_time_us":104}
I20260619 18:47:52.621655 12189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:52.606758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:41539 (local address 127.0.0.1:42502)
0619 18:47:52.607396 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:52.607414 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:52.607542 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:52.607860 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:52.607863 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:52.608588 (+ 725us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:47:52.608844 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:52.608852 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:52.610262 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:52.610266 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:52.611237 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:52.611248 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:52.612220 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:52.612225 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:52.612380 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:52.613071 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:47:52.613107 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:47:52.615960 (+ 2853us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:47:52.618731 (+ 2771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:52.618744 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:52.618752 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:52.619337 (+ 585us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:52.619682 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:52.619687 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:52.619691 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:52.619874 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:52.620521 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:47:52.620530 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:47:52.620721 (+ 191us) client_negotiation.cc:769] Sending connection context
0619 18:47:52.620905 (+ 184us) client_negotiation.cc:240] Negotiation successful
0619 18:47:52.621200 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":356,"threads_started":1}
I20260619 18:47:52.626883 11640 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.3.243.126@KRBTEST.COM'} at 127.0.0.1:42502:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 18:47:52.628101 11640 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260619 18:47:52.636689 11987 tablet.cc:2369] T c0e9d3eca12e49b4967b368bbc068b85 P 9ceae0288d68442b9b808ed536256768: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:47:52.642437 11916 tablet_service.cc:1467] Processing CreateTablet for tablet cfb298fe05cc4dbfbdbf482967e7048d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e4b992f7893b441fa2353982031efc6e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 18:47:52.643033 11916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfb298fe05cc4dbfbdbf482967e7048d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:47:52.649113 11777 tablet_service.cc:1467] Processing CreateTablet for tablet cfb298fe05cc4dbfbdbf482967e7048d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e4b992f7893b441fa2353982031efc6e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 18:47:52.649703 11777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfb298fe05cc4dbfbdbf482967e7048d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:47:52.650417 12055 tablet_service.cc:1467] Processing CreateTablet for tablet cfb298fe05cc4dbfbdbf482967e7048d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e4b992f7893b441fa2353982031efc6e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 18:47:52.650975 12055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfb298fe05cc4dbfbdbf482967e7048d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:47:52.652374 12152 tablet_bootstrap.cc:492] T cfb298fe05cc4dbfbdbf482967e7048d P 9ceae0288d68442b9b808ed536256768: Bootstrap starting.
I20260619 18:47:52.654033 12152 tablet_bootstrap.cc:654] T cfb298fe05cc4dbfbdbf482967e7048d P 9ceae0288d68442b9b808ed536256768: Neither blocks nor log segments found. Creating new log.
I20260619 18:47:52.658499 12151 tablet_bootstrap.cc:492] T cfb298fe05cc4dbfbdbf482967e7048d P a236998d0ba646d693900cf720e416a1: Bootstrap starting.
I20260619 18:47:52.660287 12150 tablet_bootstrap.cc:492] T cfb298fe05cc4dbfbdbf482967e7048d P 653a69448f244a2a99faad646667dda7: Bootstrap starting.
I20260619 18:47:52.660352 12151 tablet_bootstrap.cc:654] T cfb298fe05cc4dbfbdbf482967e7048d P a236998d0ba646d693900cf720e416a1: Neither blocks nor log segments found. Creating new log.
I20260619 18:47:52.661449 12152 tablet_bootstrap.cc:492] T cfb298fe05cc4dbfbdbf482967e7048d P 9ceae0288d68442b9b808ed536256768: No bootstrap required, opened a new log
I20260619 18:47:52.661931 12150 tablet_bootstrap.cc:654] T cfb298fe05cc4dbfbdbf482967e7048d P 653a69448f244a2a99faad646667dda7: Neither blocks nor log segments found. Creating new log.
I20260619 18:47:52.662236 12152 ts_tablet_manager.cc:1397] T cfb298fe05cc4dbfbdbf482967e7048d P 9ceae0288d68442b9b808ed536256768: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.006s
I20260619 18:47:52.664204 12152 raft_consensus.cc:348] T cfb298fe05cc4dbfbdbf482967e7048d P 9ceae0288d68442b9b808ed536256768 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653a69448f244a2a99faad646667dda7" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 35771 } } peers { permanent_uuid: "9ceae0288d68442b9b808ed536256768" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 45059 } } peers { permanent_uuid: "a236998d0ba646d693900cf720e416a1" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40189 } }
I20260619 18:47:52.664552 12152 raft_consensus.cc:374] T cfb298fe05cc4dbfbdbf482967e7048d P 9ceae0288d68442b9b808ed536256768 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:47:52.664676 12152 raft_consensus.cc:729] T cfb298fe05cc4dbfbdbf482967e7048d P 9ceae0288d68442b9b808ed536256768 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ceae0288d68442b9b808ed536256768, State: Initialized, Role: FOLLOWER
I20260619 18:47:52.665035 12152 consensus_queue.cc:260] T cfb298fe05cc4dbfbdbf482967e7048d P 9ceae0288d68442b9b808ed536256768 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653a69448f244a2a99faad646667dda7" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 35771 } } peers { permanent_uuid: "9ceae0288d68442b9b808ed536256768" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 45059 } } peers { permanent_uuid: "a236998d0ba646d693900cf720e416a1" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40189 } }
I20260619 18:47:52.665835 12152 ts_tablet_manager.cc:1428] T cfb298fe05cc4dbfbdbf482967e7048d P 9ceae0288d68442b9b808ed536256768: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260619 18:47:52.665647 12160 tablet_replica.cc:439] T cfb298fe05cc4dbfbdbf482967e7048d P 9ceae0288d68442b9b808ed536256768: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653a69448f244a2a99faad646667dda7" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 35771 } } peers { permanent_uuid: "9ceae0288d68442b9b808ed536256768" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 45059 } } peers { permanent_uuid: "a236998d0ba646d693900cf720e416a1" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40189 } } }
I20260619 18:47:52.666047 12160 tablet_replica.cc:442] T cfb298fe05cc4dbfbdbf482967e7048d P 9ceae0288d68442b9b808ed536256768: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 18:47:52.667675 12150 tablet_bootstrap.cc:492] T cfb298fe05cc4dbfbdbf482967e7048d P 653a69448f244a2a99faad646667dda7: No bootstrap required, opened a new log
I20260619 18:47:52.667841 12150 ts_tablet_manager.cc:1397] T cfb298fe05cc4dbfbdbf482967e7048d P 653a69448f244a2a99faad646667dda7: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.004s
I20260619 18:47:52.668571 12150 raft_consensus.cc:348] T cfb298fe05cc4dbfbdbf482967e7048d P 653a69448f244a2a99faad646667dda7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653a69448f244a2a99faad646667dda7" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 35771 } } peers { permanent_uuid: "9ceae0288d68442b9b808ed536256768" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 45059 } } peers { permanent_uuid: "a236998d0ba646d693900cf720e416a1" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40189 } }
I20260619 18:47:52.668810 12150 raft_consensus.cc:374] T cfb298fe05cc4dbfbdbf482967e7048d P 653a69448f244a2a99faad646667dda7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:47:52.668936 12150 raft_consensus.cc:729] T cfb298fe05cc4dbfbdbf482967e7048d P 653a69448f244a2a99faad646667dda7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 653a69448f244a2a99faad646667dda7, State: Initialized, Role: FOLLOWER
I20260619 18:47:52.669224 12150 consensus_queue.cc:260] T cfb298fe05cc4dbfbdbf482967e7048d P 653a69448f244a2a99faad646667dda7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653a69448f244a2a99faad646667dda7" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 35771 } } peers { permanent_uuid: "9ceae0288d68442b9b808ed536256768" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 45059 } } peers { permanent_uuid: "a236998d0ba646d693900cf720e416a1" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40189 } }
I20260619 18:47:52.669627 12156 tablet_replica.cc:439] T cfb298fe05cc4dbfbdbf482967e7048d P 653a69448f244a2a99faad646667dda7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653a69448f244a2a99faad646667dda7" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 35771 } } peers { permanent_uuid: "9ceae0288d68442b9b808ed536256768" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 45059 } } peers { permanent_uuid: "a236998d0ba646d693900cf720e416a1" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40189 } } }
I20260619 18:47:52.669879 12151 tablet_bootstrap.cc:492] T cfb298fe05cc4dbfbdbf482967e7048d P a236998d0ba646d693900cf720e416a1: No bootstrap required, opened a new log
I20260619 18:47:52.669886 12156 tablet_replica.cc:442] T cfb298fe05cc4dbfbdbf482967e7048d P 653a69448f244a2a99faad646667dda7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 18:47:52.670068 12151 ts_tablet_manager.cc:1397] T cfb298fe05cc4dbfbdbf482967e7048d P a236998d0ba646d693900cf720e416a1: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20260619 18:47:52.670351 12150 ts_tablet_manager.cc:1428] T cfb298fe05cc4dbfbdbf482967e7048d P 653a69448f244a2a99faad646667dda7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 18:47:52.671463 12151 raft_consensus.cc:348] T cfb298fe05cc4dbfbdbf482967e7048d P a236998d0ba646d693900cf720e416a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653a69448f244a2a99faad646667dda7" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 35771 } } peers { permanent_uuid: "9ceae0288d68442b9b808ed536256768" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 45059 } } peers { permanent_uuid: "a236998d0ba646d693900cf720e416a1" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40189 } }
I20260619 18:47:52.671674 12151 raft_consensus.cc:374] T cfb298fe05cc4dbfbdbf482967e7048d P a236998d0ba646d693900cf720e416a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:47:52.671757 12151 raft_consensus.cc:729] T cfb298fe05cc4dbfbdbf482967e7048d P a236998d0ba646d693900cf720e416a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a236998d0ba646d693900cf720e416a1, State: Initialized, Role: FOLLOWER
I20260619 18:47:52.672008 12151 consensus_queue.cc:260] T cfb298fe05cc4dbfbdbf482967e7048d P a236998d0ba646d693900cf720e416a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653a69448f244a2a99faad646667dda7" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 35771 } } peers { permanent_uuid: "9ceae0288d68442b9b808ed536256768" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 45059 } } peers { permanent_uuid: "a236998d0ba646d693900cf720e416a1" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40189 } }
I20260619 18:47:52.672463 12158 tablet_replica.cc:439] T cfb298fe05cc4dbfbdbf482967e7048d P a236998d0ba646d693900cf720e416a1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653a69448f244a2a99faad646667dda7" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 35771 } } peers { permanent_uuid: "9ceae0288d68442b9b808ed536256768" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 45059 } } peers { permanent_uuid: "a236998d0ba646d693900cf720e416a1" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40189 } } }
I20260619 18:47:52.672652 12151 ts_tablet_manager.cc:1428] T cfb298fe05cc4dbfbdbf482967e7048d P a236998d0ba646d693900cf720e416a1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 18:47:52.672765 12158 tablet_replica.cc:442] T cfb298fe05cc4dbfbdbf482967e7048d P a236998d0ba646d693900cf720e416a1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 18:47:52.750646 12156 raft_consensus.cc:482] T cfb298fe05cc4dbfbdbf482967e7048d P 653a69448f244a2a99faad646667dda7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:47:52.750876 12156 raft_consensus.cc:504] T cfb298fe05cc4dbfbdbf482967e7048d P 653a69448f244a2a99faad646667dda7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653a69448f244a2a99faad646667dda7" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 35771 } } peers { permanent_uuid: "9ceae0288d68442b9b808ed536256768" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 45059 } } peers { permanent_uuid: "a236998d0ba646d693900cf720e416a1" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40189 } }
I20260619 18:47:52.751528 12156 leader_election.cc:288] T cfb298fe05cc4dbfbdbf482967e7048d P 653a69448f244a2a99faad646667dda7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ceae0288d68442b9b808ed536256768 (127.3.243.66:45059), a236998d0ba646d693900cf720e416a1 (127.3.243.67:40189)
I20260619 18:47:52.752017 11936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfb298fe05cc4dbfbdbf482967e7048d" candidate_uuid: "653a69448f244a2a99faad646667dda7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ceae0288d68442b9b808ed536256768" is_pre_election: true
I20260619 18:47:52.752315 11936 raft_consensus.cc:2454] T cfb298fe05cc4dbfbdbf482967e7048d P 9ceae0288d68442b9b808ed536256768 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 653a69448f244a2a99faad646667dda7 in term 0.
I20260619 18:47:52.752252 12075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfb298fe05cc4dbfbdbf482967e7048d" candidate_uuid: "653a69448f244a2a99faad646667dda7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a236998d0ba646d693900cf720e416a1" is_pre_election: true
I20260619 18:47:52.752487 12075 raft_consensus.cc:2454] T cfb298fe05cc4dbfbdbf482967e7048d P a236998d0ba646d693900cf720e416a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 653a69448f244a2a99faad646667dda7 in term 0.
I20260619 18:47:52.752784 11730 leader_election.cc:302] T cfb298fe05cc4dbfbdbf482967e7048d P 653a69448f244a2a99faad646667dda7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 653a69448f244a2a99faad646667dda7, 9ceae0288d68442b9b808ed536256768; no voters:
I20260619 18:47:52.753057 12156 raft_consensus.cc:2781] T cfb298fe05cc4dbfbdbf482967e7048d P 653a69448f244a2a99faad646667dda7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:47:52.753176 12156 raft_consensus.cc:482] T cfb298fe05cc4dbfbdbf482967e7048d P 653a69448f244a2a99faad646667dda7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:47:52.753285 12156 raft_consensus.cc:3037] T cfb298fe05cc4dbfbdbf482967e7048d P 653a69448f244a2a99faad646667dda7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:47:52.756103 12156 raft_consensus.cc:504] T cfb298fe05cc4dbfbdbf482967e7048d P 653a69448f244a2a99faad646667dda7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653a69448f244a2a99faad646667dda7" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 35771 } } peers { permanent_uuid: "9ceae0288d68442b9b808ed536256768" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 45059 } } peers { permanent_uuid: "a236998d0ba646d693900cf720e416a1" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40189 } }
I20260619 18:47:52.756733 12156 leader_election.cc:288] T cfb298fe05cc4dbfbdbf482967e7048d P 653a69448f244a2a99faad646667dda7 [CANDIDATE]: Term 1 election: Requested vote from peers 9ceae0288d68442b9b808ed536256768 (127.3.243.66:45059), a236998d0ba646d693900cf720e416a1 (127.3.243.67:40189)
I20260619 18:47:52.757109 11936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfb298fe05cc4dbfbdbf482967e7048d" candidate_uuid: "653a69448f244a2a99faad646667dda7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ceae0288d68442b9b808ed536256768"
I20260619 18:47:52.757174 12075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfb298fe05cc4dbfbdbf482967e7048d" candidate_uuid: "653a69448f244a2a99faad646667dda7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a236998d0ba646d693900cf720e416a1"
I20260619 18:47:52.757311 11936 raft_consensus.cc:3037] T cfb298fe05cc4dbfbdbf482967e7048d P 9ceae0288d68442b9b808ed536256768 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:47:52.757366 12075 raft_consensus.cc:3037] T cfb298fe05cc4dbfbdbf482967e7048d P a236998d0ba646d693900cf720e416a1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:47:52.759344 12075 raft_consensus.cc:2454] T cfb298fe05cc4dbfbdbf482967e7048d P a236998d0ba646d693900cf720e416a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 653a69448f244a2a99faad646667dda7 in term 1.
I20260619 18:47:52.759681 11728 leader_election.cc:302] T cfb298fe05cc4dbfbdbf482967e7048d P 653a69448f244a2a99faad646667dda7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 653a69448f244a2a99faad646667dda7, a236998d0ba646d693900cf720e416a1; no voters:
I20260619 18:47:52.759923 12156 raft_consensus.cc:2781] T cfb298fe05cc4dbfbdbf482967e7048d P 653a69448f244a2a99faad646667dda7 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:47:52.760095 12156 raft_consensus.cc:686] T cfb298fe05cc4dbfbdbf482967e7048d P 653a69448f244a2a99faad646667dda7 [term 1 LEADER]: Becoming Leader. State: Replica: 653a69448f244a2a99faad646667dda7, State: Running, Role: LEADER
I20260619 18:47:52.760115 11936 raft_consensus.cc:2454] T cfb298fe05cc4dbfbdbf482967e7048d P 9ceae0288d68442b9b808ed536256768 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 653a69448f244a2a99faad646667dda7 in term 1.
I20260619 18:47:52.760372 12156 consensus_queue.cc:237] T cfb298fe05cc4dbfbdbf482967e7048d P 653a69448f244a2a99faad646667dda7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653a69448f244a2a99faad646667dda7" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 35771 } } peers { permanent_uuid: "9ceae0288d68442b9b808ed536256768" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 45059 } } peers { permanent_uuid: "a236998d0ba646d693900cf720e416a1" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40189 } }
I20260619 18:47:52.761973 12182 tablet_replica.cc:439] T cfb298fe05cc4dbfbdbf482967e7048d P 653a69448f244a2a99faad646667dda7: TxnStatusTablet state changed. Reason: New leader 653a69448f244a2a99faad646667dda7. Latest consensus state: current_term: 1 leader_uuid: "653a69448f244a2a99faad646667dda7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653a69448f244a2a99faad646667dda7" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 35771 } } peers { permanent_uuid: "9ceae0288d68442b9b808ed536256768" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 45059 } } peers { permanent_uuid: "a236998d0ba646d693900cf720e416a1" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40189 } } }
I20260619 18:47:52.762300 12182 tablet_replica.cc:442] T cfb298fe05cc4dbfbdbf482967e7048d P 653a69448f244a2a99faad646667dda7: This TxnStatusTablet replica's current role is: LEADER
I20260619 18:47:52.762887 12194 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 18:47:52.763057 12194 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 18:47:52.763507 11639 catalog_manager.cc:5495] T cfb298fe05cc4dbfbdbf482967e7048d P 653a69448f244a2a99faad646667dda7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 653a69448f244a2a99faad646667dda7 (127.3.243.65). New cstate: current_term: 1 leader_uuid: "653a69448f244a2a99faad646667dda7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653a69448f244a2a99faad646667dda7" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 35771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ceae0288d68442b9b808ed536256768" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 45059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a236998d0ba646d693900cf720e416a1" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40189 } health_report { overall_health: UNKNOWN } } }
I20260619 18:47:52.786016 12197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:52.779355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39296 (local address 127.3.243.65:35771)
0619 18:47:52.779973 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:52.779977 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:52.779994 (+ 17us) server_negotiation.cc:407] Connection header received
0619 18:47:52.780062 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:52.780065 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:52.780206 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:52.780372 (+ 166us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:47:52.780793 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:52.781615 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:52.783515 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:52.784246 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:52.784353 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:52.784845 (+ 492us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:47:52.785433 (+ 588us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:47:52.785565 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:52.785684 (+ 119us) server_negotiation.cc:299] Negotiation successful
0619 18:47:52.785748 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":501,"thread_start_us":335,"threads_started":1}
I20260619 18:47:52.786114 12189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:52.779166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.65:35771 (local address 127.0.0.1:39296)
0619 18:47:52.779378 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:52.779397 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:52.779496 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:52.780419 (+ 923us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:52.780425 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:52.780439 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 18:47:52.780639 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:52.780650 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:52.781746 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:52.781752 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:52.783350 (+ 1598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:52.783365 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:52.784376 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:52.784383 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:52.784576 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:52.784649 (+ 73us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 18:47:52.785548 (+ 899us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 18:47:52.785571 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 18:47:52.785663 (+ 92us) client_negotiation.cc:240] Negotiation successful
0619 18:47:52.785819 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260619 18:47:52.790217 11936 raft_consensus.cc:1261] T cfb298fe05cc4dbfbdbf482967e7048d P 9ceae0288d68442b9b808ed536256768 [term 1 FOLLOWER]: Refusing update from remote peer 653a69448f244a2a99faad646667dda7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:47:52.790390 12075 raft_consensus.cc:1261] T cfb298fe05cc4dbfbdbf482967e7048d P a236998d0ba646d693900cf720e416a1 [term 1 FOLLOWER]: Refusing update from remote peer 653a69448f244a2a99faad646667dda7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:47:52.790936 12182 consensus_queue.cc:1035] T cfb298fe05cc4dbfbdbf482967e7048d P 653a69448f244a2a99faad646667dda7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ceae0288d68442b9b808ed536256768" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 45059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:47:52.791247 12156 consensus_queue.cc:1035] T cfb298fe05cc4dbfbdbf482967e7048d P 653a69448f244a2a99faad646667dda7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a236998d0ba646d693900cf720e416a1" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:47:52.801060 12160 tablet_replica.cc:439] T cfb298fe05cc4dbfbdbf482967e7048d P 9ceae0288d68442b9b808ed536256768: TxnStatusTablet state changed. Reason: New leader 653a69448f244a2a99faad646667dda7. Latest consensus state: current_term: 1 leader_uuid: "653a69448f244a2a99faad646667dda7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653a69448f244a2a99faad646667dda7" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 35771 } } peers { permanent_uuid: "9ceae0288d68442b9b808ed536256768" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 45059 } } peers { permanent_uuid: "a236998d0ba646d693900cf720e416a1" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40189 } } }
I20260619 18:47:52.801394 12160 tablet_replica.cc:442] T cfb298fe05cc4dbfbdbf482967e7048d P 9ceae0288d68442b9b808ed536256768: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 18:47:52.803071 12182 tablet_replica.cc:439] T cfb298fe05cc4dbfbdbf482967e7048d P 653a69448f244a2a99faad646667dda7: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "653a69448f244a2a99faad646667dda7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653a69448f244a2a99faad646667dda7" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 35771 } } peers { permanent_uuid: "9ceae0288d68442b9b808ed536256768" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 45059 } } peers { permanent_uuid: "a236998d0ba646d693900cf720e416a1" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40189 } } }
I20260619 18:47:52.803335 12182 tablet_replica.cc:442] T cfb298fe05cc4dbfbdbf482967e7048d P 653a69448f244a2a99faad646667dda7: This TxnStatusTablet replica's current role is: LEADER
I20260619 18:47:52.803941 12166 mvcc.cc:205] Tried to move back new op lower bound from 7298641398941835264 to 7298641398828957696. Current Snapshot: MvccSnapshot[applied={T|T < 7298641398941835264}]
I20260619 18:47:52.805316 12165 mvcc.cc:205] Tried to move back new op lower bound from 7298641398941835264 to 7298641398828957696. Current Snapshot: MvccSnapshot[applied={T|T < 7298641398941835264}]
I20260619 18:47:52.805593 12160 tablet_replica.cc:439] T cfb298fe05cc4dbfbdbf482967e7048d P 9ceae0288d68442b9b808ed536256768: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "653a69448f244a2a99faad646667dda7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653a69448f244a2a99faad646667dda7" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 35771 } } peers { permanent_uuid: "9ceae0288d68442b9b808ed536256768" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 45059 } } peers { permanent_uuid: "a236998d0ba646d693900cf720e416a1" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40189 } } }
I20260619 18:47:52.805792 12160 tablet_replica.cc:442] T cfb298fe05cc4dbfbdbf482967e7048d P 9ceae0288d68442b9b808ed536256768: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 18:47:52.806030 12158 tablet_replica.cc:439] T cfb298fe05cc4dbfbdbf482967e7048d P a236998d0ba646d693900cf720e416a1: TxnStatusTablet state changed. Reason: New leader 653a69448f244a2a99faad646667dda7. Latest consensus state: current_term: 1 leader_uuid: "653a69448f244a2a99faad646667dda7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653a69448f244a2a99faad646667dda7" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 35771 } } peers { permanent_uuid: "9ceae0288d68442b9b808ed536256768" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 45059 } } peers { permanent_uuid: "a236998d0ba646d693900cf720e416a1" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40189 } } }
I20260619 18:47:52.806319 12158 tablet_replica.cc:442] T cfb298fe05cc4dbfbdbf482967e7048d P a236998d0ba646d693900cf720e416a1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 18:47:52.807170 12156 tablet_replica.cc:439] T cfb298fe05cc4dbfbdbf482967e7048d P 653a69448f244a2a99faad646667dda7: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "653a69448f244a2a99faad646667dda7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653a69448f244a2a99faad646667dda7" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 35771 } } peers { permanent_uuid: "9ceae0288d68442b9b808ed536256768" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 45059 } } peers { permanent_uuid: "a236998d0ba646d693900cf720e416a1" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40189 } } }
I20260619 18:47:52.807467 12156 tablet_replica.cc:442] T cfb298fe05cc4dbfbdbf482967e7048d P 653a69448f244a2a99faad646667dda7: This TxnStatusTablet replica's current role is: LEADER
I20260619 18:47:52.810210 12158 tablet_replica.cc:439] T cfb298fe05cc4dbfbdbf482967e7048d P a236998d0ba646d693900cf720e416a1: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "653a69448f244a2a99faad646667dda7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653a69448f244a2a99faad646667dda7" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 35771 } } peers { permanent_uuid: "9ceae0288d68442b9b808ed536256768" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 45059 } } peers { permanent_uuid: "a236998d0ba646d693900cf720e416a1" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40189 } } }
I20260619 18:47:52.810444 12158 tablet_replica.cc:442] T cfb298fe05cc4dbfbdbf482967e7048d P a236998d0ba646d693900cf720e416a1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 18:47:52.819123 12197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:52.814239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39302 (local address 127.3.243.65:35771)
0619 18:47:52.814381 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:52.814388 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:52.814740 (+ 352us) server_negotiation.cc:407] Connection header received
0619 18:47:52.814916 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:52.814921 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:52.814988 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:52.815058 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:47:52.815483 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:52.816278 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:52.817560 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:52.817980 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:52.818091 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:52.818474 (+ 383us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:47:52.818568 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:47:52.818643 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:52.818764 (+ 121us) server_negotiation.cc:299] Negotiation successful
0619 18:47:52.818835 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 18:47:52.834077 12197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:52.828738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.65:47791 (local address 127.3.243.65:35771)
0619 18:47:52.828932 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:52.828938 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:52.829512 (+ 574us) server_negotiation.cc:407] Connection header received
0619 18:47:52.829683 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:52.829688 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:52.829772 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:52.829841 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:47:52.830338 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:52.831102 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:52.832495 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:52.832918 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:52.833017 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:52.833409 (+ 392us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:47:52.833497 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:47:52.833562 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:52.833682 (+ 120us) server_negotiation.cc:299] Negotiation successful
0619 18:47:52.833745 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 18:47:52.834153 12204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:52.828753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.65:35771 (local address 127.3.243.65:47791)
0619 18:47:52.829428 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:52.829442 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:52.829573 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:52.829864 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:52.829869 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:52.829879 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 18:47:52.830195 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:52.830203 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:52.831213 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:52.831218 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:52.832381 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:52.832392 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:52.833076 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:52.833082 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:52.833213 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:52.833259 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 18:47:52.833576 (+ 317us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 18:47:52.833586 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 18:47:52.833655 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 18:47:52.833788 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":391,"threads_started":1}
I20260619 18:47:52.842850 12075 raft_consensus.cc:1261] T c0e9d3eca12e49b4967b368bbc068b85 P a236998d0ba646d693900cf720e416a1 [term 1 FOLLOWER]: Refusing update from remote peer 653a69448f244a2a99faad646667dda7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:47:52.842983 11936 raft_consensus.cc:1261] T c0e9d3eca12e49b4967b368bbc068b85 P 9ceae0288d68442b9b808ed536256768 [term 1 FOLLOWER]: Refusing update from remote peer 653a69448f244a2a99faad646667dda7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:47:52.843537 12182 consensus_queue.cc:1035] T c0e9d3eca12e49b4967b368bbc068b85 P 653a69448f244a2a99faad646667dda7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ceae0288d68442b9b808ed536256768" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 45059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:47:52.843991 12156 consensus_queue.cc:1035] T c0e9d3eca12e49b4967b368bbc068b85 P 653a69448f244a2a99faad646667dda7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a236998d0ba646d693900cf720e416a1" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:47:52.853760 12164 mvcc.cc:205] Tried to move back new op lower bound from 7298641399160078336 to 7298641397914439680. Current Snapshot: MvccSnapshot[applied={T|T < 7298641399160078336}]
I20260619 18:47:52.900319 11640 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42488:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:47:52.900528 11640 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42488:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:47:52.903545 11640 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 022c3c6be95b46f8bf8e526a33780f6d: Sending DeleteTablet for 3 replicas of tablet c0e9d3eca12e49b4967b368bbc068b85
I20260619 18:47:52.904323 11916 tablet_service.cc:1514] Processing DeleteTablet for tablet c0e9d3eca12e49b4967b368bbc068b85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:47:52 UTC) from {username='oryx', principal='oryx/127.3.243.126@KRBTEST.COM'} at 127.0.0.1:38872
I20260619 18:47:52.904428 12055 tablet_service.cc:1514] Processing DeleteTablet for tablet c0e9d3eca12e49b4967b368bbc068b85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:47:52 UTC) from {username='oryx', principal='oryx/127.3.243.126@KRBTEST.COM'} at 127.0.0.1:33140
I20260619 18:47:52.904595 12174 tablet_replica.cc:330] T c0e9d3eca12e49b4967b368bbc068b85 P 9ceae0288d68442b9b808ed536256768: stopping tablet replica
I20260619 18:47:52.904790 12174 raft_consensus.cc:2229] T c0e9d3eca12e49b4967b368bbc068b85 P 9ceae0288d68442b9b808ed536256768 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:47:52.904923 12209 tablet_replica.cc:330] T c0e9d3eca12e49b4967b368bbc068b85 P a236998d0ba646d693900cf720e416a1: stopping tablet replica
I20260619 18:47:52.904965 12174 raft_consensus.cc:2258] T c0e9d3eca12e49b4967b368bbc068b85 P 9ceae0288d68442b9b808ed536256768 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:47:52.905108 12209 raft_consensus.cc:2229] T c0e9d3eca12e49b4967b368bbc068b85 P a236998d0ba646d693900cf720e416a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:47:52.905298 12209 raft_consensus.cc:2258] T c0e9d3eca12e49b4967b368bbc068b85 P a236998d0ba646d693900cf720e416a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:47:52.905560 11776 tablet_service.cc:1514] Processing DeleteTablet for tablet c0e9d3eca12e49b4967b368bbc068b85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:47:52 UTC) from {username='oryx', principal='oryx/127.3.243.126@KRBTEST.COM'} at 127.0.0.1:39282
I20260619 18:47:52.905958 12174 ts_tablet_manager.cc:1905] T c0e9d3eca12e49b4967b368bbc068b85 P 9ceae0288d68442b9b808ed536256768: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:47:52.906328 12209 ts_tablet_manager.cc:1905] T c0e9d3eca12e49b4967b368bbc068b85 P a236998d0ba646d693900cf720e416a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:47:52.912513 12174 ts_tablet_manager.cc:1918] T c0e9d3eca12e49b4967b368bbc068b85 P 9ceae0288d68442b9b808ed536256768: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 18:47:52.912665 12174 log.cc:1196] T c0e9d3eca12e49b4967b368bbc068b85 P 9ceae0288d68442b9b808ed536256768: Deleting WAL directory at /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/minicluster-data/ts-1/wal/wals/c0e9d3eca12e49b4967b368bbc068b85
I20260619 18:47:52.913070 12174 ts_tablet_manager.cc:1939] T c0e9d3eca12e49b4967b368bbc068b85 P 9ceae0288d68442b9b808ed536256768: Deleting consensus metadata
I20260619 18:47:52.913271 12209 ts_tablet_manager.cc:1918] T c0e9d3eca12e49b4967b368bbc068b85 P a236998d0ba646d693900cf720e416a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 18:47:52.913400 12209 log.cc:1196] T c0e9d3eca12e49b4967b368bbc068b85 P a236998d0ba646d693900cf720e416a1: Deleting WAL directory at /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/minicluster-data/ts-2/wal/wals/c0e9d3eca12e49b4967b368bbc068b85
I20260619 18:47:52.913941 12209 ts_tablet_manager.cc:1939] T c0e9d3eca12e49b4967b368bbc068b85 P a236998d0ba646d693900cf720e416a1: Deleting consensus metadata
I20260619 18:47:52.914330 11627 catalog_manager.cc:4841] TS 9ceae0288d68442b9b808ed536256768 (127.3.243.66:45059): tablet c0e9d3eca12e49b4967b368bbc068b85 (table test-table [id=8e87f934554744baa7bb092e2bf348b4]) successfully deleted
I20260619 18:47:52.915006 11625 catalog_manager.cc:4841] TS a236998d0ba646d693900cf720e416a1 (127.3.243.67:40189): tablet c0e9d3eca12e49b4967b368bbc068b85 (table test-table [id=8e87f934554744baa7bb092e2bf348b4]) successfully deleted
I20260619 18:47:52.917343 12210 tablet_replica.cc:330] T c0e9d3eca12e49b4967b368bbc068b85 P 653a69448f244a2a99faad646667dda7: stopping tablet replica
I20260619 18:47:52.917536 12210 raft_consensus.cc:2229] T c0e9d3eca12e49b4967b368bbc068b85 P 653a69448f244a2a99faad646667dda7 [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:47:52.918090 12210 raft_consensus.cc:2258] T c0e9d3eca12e49b4967b368bbc068b85 P 653a69448f244a2a99faad646667dda7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:47:52.919025 12210 ts_tablet_manager.cc:1905] T c0e9d3eca12e49b4967b368bbc068b85 P 653a69448f244a2a99faad646667dda7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:47:52.926499 12210 ts_tablet_manager.cc:1918] T c0e9d3eca12e49b4967b368bbc068b85 P 653a69448f244a2a99faad646667dda7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 18:47:52.926640 12210 log.cc:1196] T c0e9d3eca12e49b4967b368bbc068b85 P 653a69448f244a2a99faad646667dda7: Deleting WAL directory at /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/minicluster-data/ts-0/wal/wals/c0e9d3eca12e49b4967b368bbc068b85
I20260619 18:47:52.927084 12210 ts_tablet_manager.cc:1939] T c0e9d3eca12e49b4967b368bbc068b85 P 653a69448f244a2a99faad646667dda7: Deleting consensus metadata
I20260619 18:47:52.928232 11625 catalog_manager.cc:4841] TS 653a69448f244a2a99faad646667dda7 (127.3.243.65:35771): tablet c0e9d3eca12e49b4967b368bbc068b85 (table test-table [id=8e87f934554744baa7bb092e2bf348b4]) successfully deleted
Jun 19 18:47:52 dist-test-slave-m9wg krb5kdc[11595](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894872, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 18:47:52 dist-test-slave-m9wg krb5kdc[11595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.3.243.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:47:52 dist-test-slave-m9wg krb5kdc[11595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.3.243.126@KRBTEST.COM, Server not found in Kerberos database
W20260619 18:47:52.960700 11735 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:47:52.950440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42506 (local address 127.3.243.126:41539)
0619 18:47:52.950605 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:52.950610 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:52.951025 (+ 415us) server_negotiation.cc:407] Connection header received
0619 18:47:52.951219 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:52.951225 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:52.951298 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:52.951381 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:52.952690 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:52.953381 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:52.954677 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:52.955234 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:52.955342 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:52.960454 (+ 5112us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42506: BlockingRecv error: recv got EOF from 127.0.0.1:42506 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Jun 19 18:47:52 dist-test-slave-m9wg krb5kdc[11595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894872, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM
I20260619 18:47:52.983516 11735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:52.969610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42514 (local address 127.3.243.126:41539)
0619 18:47:52.969800 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:52.969807 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:52.970297 (+ 490us) server_negotiation.cc:407] Connection header received
0619 18:47:52.970498 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:52.970505 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:52.970576 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:52.970660 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:52.972009 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:52.972777 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:52.973963 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:52.974367 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:52.974470 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:52.978496 (+ 4026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:52.978501 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:52.978504 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:52.978559 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:52.981053 (+ 2494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:52.981810 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:52.981814 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:52.981816 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:52.981934 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:52.982284 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:52.982287 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:52.982289 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:52.982542 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:52.982720 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:52.983048 (+ 328us) server_negotiation.cc:299] Negotiation successful
0619 18:47:52.983240 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 18:47:52.986791 11640 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42514:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:47:52.987386 11640 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:47:52.993800 11776 tablet_service.cc:1467] Processing CreateTablet for tablet 3d2edde41e714ca9842d98a4b891384f (DEFAULT_TABLE table=test-table [id=3773263a0ffa4c3c9e7137278f645f8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:47:52.994091 12055 tablet_service.cc:1467] Processing CreateTablet for tablet 3d2edde41e714ca9842d98a4b891384f (DEFAULT_TABLE table=test-table [id=3773263a0ffa4c3c9e7137278f645f8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:47:52.994091 11916 tablet_service.cc:1467] Processing CreateTablet for tablet 3d2edde41e714ca9842d98a4b891384f (DEFAULT_TABLE table=test-table [id=3773263a0ffa4c3c9e7137278f645f8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:47:52.994295 11776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d2edde41e714ca9842d98a4b891384f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:47:52.994504 11916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d2edde41e714ca9842d98a4b891384f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:47:52.994495 12055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d2edde41e714ca9842d98a4b891384f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:47:53.000630 12152 tablet_bootstrap.cc:492] T 3d2edde41e714ca9842d98a4b891384f P 9ceae0288d68442b9b808ed536256768: Bootstrap starting.
I20260619 18:47:53.000622 12151 tablet_bootstrap.cc:492] T 3d2edde41e714ca9842d98a4b891384f P a236998d0ba646d693900cf720e416a1: Bootstrap starting.
I20260619 18:47:53.000736 12150 tablet_bootstrap.cc:492] T 3d2edde41e714ca9842d98a4b891384f P 653a69448f244a2a99faad646667dda7: Bootstrap starting.
I20260619 18:47:53.002156 12152 tablet_bootstrap.cc:654] T 3d2edde41e714ca9842d98a4b891384f P 9ceae0288d68442b9b808ed536256768: Neither blocks nor log segments found. Creating new log.
I20260619 18:47:53.002174 12151 tablet_bootstrap.cc:654] T 3d2edde41e714ca9842d98a4b891384f P a236998d0ba646d693900cf720e416a1: Neither blocks nor log segments found. Creating new log.
I20260619 18:47:53.002200 12150 tablet_bootstrap.cc:654] T 3d2edde41e714ca9842d98a4b891384f P 653a69448f244a2a99faad646667dda7: Neither blocks nor log segments found. Creating new log.
I20260619 18:47:53.012457 12150 tablet_bootstrap.cc:492] T 3d2edde41e714ca9842d98a4b891384f P 653a69448f244a2a99faad646667dda7: No bootstrap required, opened a new log
I20260619 18:47:53.012692 12150 ts_tablet_manager.cc:1397] T 3d2edde41e714ca9842d98a4b891384f P 653a69448f244a2a99faad646667dda7: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.008s
I20260619 18:47:53.013561 12151 tablet_bootstrap.cc:492] T 3d2edde41e714ca9842d98a4b891384f P a236998d0ba646d693900cf720e416a1: No bootstrap required, opened a new log
I20260619 18:47:53.013760 12151 ts_tablet_manager.cc:1397] T 3d2edde41e714ca9842d98a4b891384f P a236998d0ba646d693900cf720e416a1: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.001s
I20260619 18:47:53.013767 12150 raft_consensus.cc:348] T 3d2edde41e714ca9842d98a4b891384f P 653a69448f244a2a99faad646667dda7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653a69448f244a2a99faad646667dda7" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 35771 } } peers { permanent_uuid: "9ceae0288d68442b9b808ed536256768" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 45059 } } peers { permanent_uuid: "a236998d0ba646d693900cf720e416a1" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40189 } }
I20260619 18:47:53.014190 12150 raft_consensus.cc:374] T 3d2edde41e714ca9842d98a4b891384f P 653a69448f244a2a99faad646667dda7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:47:53.014335 12150 raft_consensus.cc:729] T 3d2edde41e714ca9842d98a4b891384f P 653a69448f244a2a99faad646667dda7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 653a69448f244a2a99faad646667dda7, State: Initialized, Role: FOLLOWER
I20260619 18:47:53.014605 12151 raft_consensus.cc:348] T 3d2edde41e714ca9842d98a4b891384f P a236998d0ba646d693900cf720e416a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653a69448f244a2a99faad646667dda7" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 35771 } } peers { permanent_uuid: "9ceae0288d68442b9b808ed536256768" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 45059 } } peers { permanent_uuid: "a236998d0ba646d693900cf720e416a1" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40189 } }
I20260619 18:47:53.014851 12151 raft_consensus.cc:374] T 3d2edde41e714ca9842d98a4b891384f P a236998d0ba646d693900cf720e416a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:47:53.014948 12151 raft_consensus.cc:729] T 3d2edde41e714ca9842d98a4b891384f P a236998d0ba646d693900cf720e416a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a236998d0ba646d693900cf720e416a1, State: Initialized, Role: FOLLOWER
I20260619 18:47:53.014696 12150 consensus_queue.cc:260] T 3d2edde41e714ca9842d98a4b891384f P 653a69448f244a2a99faad646667dda7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653a69448f244a2a99faad646667dda7" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 35771 } } peers { permanent_uuid: "9ceae0288d68442b9b808ed536256768" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 45059 } } peers { permanent_uuid: "a236998d0ba646d693900cf720e416a1" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40189 } }
I20260619 18:47:53.015424 12152 tablet_bootstrap.cc:492] T 3d2edde41e714ca9842d98a4b891384f P 9ceae0288d68442b9b808ed536256768: No bootstrap required, opened a new log
I20260619 18:47:53.015570 12152 ts_tablet_manager.cc:1397] T 3d2edde41e714ca9842d98a4b891384f P 9ceae0288d68442b9b808ed536256768: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.004s
I20260619 18:47:53.015712 12150 ts_tablet_manager.cc:1428] T 3d2edde41e714ca9842d98a4b891384f P 653a69448f244a2a99faad646667dda7: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260619 18:47:53.015200 12151 consensus_queue.cc:260] T 3d2edde41e714ca9842d98a4b891384f P a236998d0ba646d693900cf720e416a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653a69448f244a2a99faad646667dda7" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 35771 } } peers { permanent_uuid: "9ceae0288d68442b9b808ed536256768" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 45059 } } peers { permanent_uuid: "a236998d0ba646d693900cf720e416a1" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40189 } }
I20260619 18:47:53.017097 12151 ts_tablet_manager.cc:1428] T 3d2edde41e714ca9842d98a4b891384f P a236998d0ba646d693900cf720e416a1: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260619 18:47:53.017935 12152 raft_consensus.cc:348] T 3d2edde41e714ca9842d98a4b891384f P 9ceae0288d68442b9b808ed536256768 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653a69448f244a2a99faad646667dda7" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 35771 } } peers { permanent_uuid: "9ceae0288d68442b9b808ed536256768" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 45059 } } peers { permanent_uuid: "a236998d0ba646d693900cf720e416a1" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40189 } }
I20260619 18:47:53.018188 12152 raft_consensus.cc:374] T 3d2edde41e714ca9842d98a4b891384f P 9ceae0288d68442b9b808ed536256768 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:47:53.018283 12152 raft_consensus.cc:729] T 3d2edde41e714ca9842d98a4b891384f P 9ceae0288d68442b9b808ed536256768 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ceae0288d68442b9b808ed536256768, State: Initialized, Role: FOLLOWER
I20260619 18:47:53.018502 12152 consensus_queue.cc:260] T 3d2edde41e714ca9842d98a4b891384f P 9ceae0288d68442b9b808ed536256768 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653a69448f244a2a99faad646667dda7" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 35771 } } peers { permanent_uuid: "9ceae0288d68442b9b808ed536256768" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 45059 } } peers { permanent_uuid: "a236998d0ba646d693900cf720e416a1" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40189 } }
I20260619 18:47:53.021188 12152 ts_tablet_manager.cc:1428] T 3d2edde41e714ca9842d98a4b891384f P 9ceae0288d68442b9b808ed536256768: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260619 18:47:53.188773 12158 raft_consensus.cc:482] T 3d2edde41e714ca9842d98a4b891384f P a236998d0ba646d693900cf720e416a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:47:53.189021 12158 raft_consensus.cc:504] T 3d2edde41e714ca9842d98a4b891384f P a236998d0ba646d693900cf720e416a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653a69448f244a2a99faad646667dda7" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 35771 } } peers { permanent_uuid: "9ceae0288d68442b9b808ed536256768" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 45059 } } peers { permanent_uuid: "a236998d0ba646d693900cf720e416a1" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40189 } }
I20260619 18:47:53.189718 12158 leader_election.cc:288] T 3d2edde41e714ca9842d98a4b891384f P a236998d0ba646d693900cf720e416a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 653a69448f244a2a99faad646667dda7 (127.3.243.65:35771), 9ceae0288d68442b9b808ed536256768 (127.3.243.66:45059)
I20260619 18:47:53.190316 11797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d2edde41e714ca9842d98a4b891384f" candidate_uuid: "a236998d0ba646d693900cf720e416a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "653a69448f244a2a99faad646667dda7" is_pre_election: true
I20260619 18:47:53.190310 11936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d2edde41e714ca9842d98a4b891384f" candidate_uuid: "a236998d0ba646d693900cf720e416a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ceae0288d68442b9b808ed536256768" is_pre_election: true
I20260619 18:47:53.190574 11797 raft_consensus.cc:2454] T 3d2edde41e714ca9842d98a4b891384f P 653a69448f244a2a99faad646667dda7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a236998d0ba646d693900cf720e416a1 in term 0.
I20260619 18:47:53.190557 11936 raft_consensus.cc:2454] T 3d2edde41e714ca9842d98a4b891384f P 9ceae0288d68442b9b808ed536256768 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a236998d0ba646d693900cf720e416a1 in term 0.
I20260619 18:47:53.190991 12007 leader_election.cc:302] T 3d2edde41e714ca9842d98a4b891384f P a236998d0ba646d693900cf720e416a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 653a69448f244a2a99faad646667dda7, a236998d0ba646d693900cf720e416a1; no voters:
I20260619 18:47:53.191251 12158 raft_consensus.cc:2781] T 3d2edde41e714ca9842d98a4b891384f P a236998d0ba646d693900cf720e416a1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:47:53.191354 12158 raft_consensus.cc:482] T 3d2edde41e714ca9842d98a4b891384f P a236998d0ba646d693900cf720e416a1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:47:53.191440 12158 raft_consensus.cc:3037] T 3d2edde41e714ca9842d98a4b891384f P a236998d0ba646d693900cf720e416a1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:47:53.193404 12158 raft_consensus.cc:504] T 3d2edde41e714ca9842d98a4b891384f P a236998d0ba646d693900cf720e416a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653a69448f244a2a99faad646667dda7" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 35771 } } peers { permanent_uuid: "9ceae0288d68442b9b808ed536256768" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 45059 } } peers { permanent_uuid: "a236998d0ba646d693900cf720e416a1" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40189 } }
I20260619 18:47:53.193926 12158 leader_election.cc:288] T 3d2edde41e714ca9842d98a4b891384f P a236998d0ba646d693900cf720e416a1 [CANDIDATE]: Term 1 election: Requested vote from peers 653a69448f244a2a99faad646667dda7 (127.3.243.65:35771), 9ceae0288d68442b9b808ed536256768 (127.3.243.66:45059)
I20260619 18:47:53.194306 11797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d2edde41e714ca9842d98a4b891384f" candidate_uuid: "a236998d0ba646d693900cf720e416a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "653a69448f244a2a99faad646667dda7"
I20260619 18:47:53.194485 11797 raft_consensus.cc:3037] T 3d2edde41e714ca9842d98a4b891384f P 653a69448f244a2a99faad646667dda7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:47:53.194444 11936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d2edde41e714ca9842d98a4b891384f" candidate_uuid: "a236998d0ba646d693900cf720e416a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ceae0288d68442b9b808ed536256768"
I20260619 18:47:53.194644 11936 raft_consensus.cc:3037] T 3d2edde41e714ca9842d98a4b891384f P 9ceae0288d68442b9b808ed536256768 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:47:53.196578 11797 raft_consensus.cc:2454] T 3d2edde41e714ca9842d98a4b891384f P 653a69448f244a2a99faad646667dda7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a236998d0ba646d693900cf720e416a1 in term 1.
I20260619 18:47:53.196926 12007 leader_election.cc:302] T 3d2edde41e714ca9842d98a4b891384f P a236998d0ba646d693900cf720e416a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 653a69448f244a2a99faad646667dda7, a236998d0ba646d693900cf720e416a1; no voters:
I20260619 18:47:53.197139 12158 raft_consensus.cc:2781] T 3d2edde41e714ca9842d98a4b891384f P a236998d0ba646d693900cf720e416a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:47:53.197512 12158 raft_consensus.cc:686] T 3d2edde41e714ca9842d98a4b891384f P a236998d0ba646d693900cf720e416a1 [term 1 LEADER]: Becoming Leader. State: Replica: a236998d0ba646d693900cf720e416a1, State: Running, Role: LEADER
I20260619 18:47:53.197522 11936 raft_consensus.cc:2454] T 3d2edde41e714ca9842d98a4b891384f P 9ceae0288d68442b9b808ed536256768 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a236998d0ba646d693900cf720e416a1 in term 1.
I20260619 18:47:53.197783 12158 consensus_queue.cc:237] T 3d2edde41e714ca9842d98a4b891384f P a236998d0ba646d693900cf720e416a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653a69448f244a2a99faad646667dda7" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 35771 } } peers { permanent_uuid: "9ceae0288d68442b9b808ed536256768" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 45059 } } peers { permanent_uuid: "a236998d0ba646d693900cf720e416a1" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40189 } }
I20260619 18:47:53.200865 11639 catalog_manager.cc:5495] T 3d2edde41e714ca9842d98a4b891384f P a236998d0ba646d693900cf720e416a1 reported cstate change: term changed from 0 to 1, leader changed from <none> to a236998d0ba646d693900cf720e416a1 (127.3.243.67). New cstate: current_term: 1 leader_uuid: "a236998d0ba646d693900cf720e416a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653a69448f244a2a99faad646667dda7" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 35771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ceae0288d68442b9b808ed536256768" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 45059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a236998d0ba646d693900cf720e416a1" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40189 } health_report { overall_health: HEALTHY } } }
I20260619 18:47:53.255805 12229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:53.251028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33158 (local address 127.3.243.67:40189)
0619 18:47:53.251595 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:53.251599 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:53.251622 (+ 23us) server_negotiation.cc:407] Connection header received
0619 18:47:53.251683 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:53.251687 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:53.251747 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:53.251831 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:47:53.252221 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:53.252941 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:53.254222 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:53.254647 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:53.254753 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:53.255130 (+ 377us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:47:53.255226 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:47:53.255297 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:53.255447 (+ 150us) server_negotiation.cc:299] Negotiation successful
0619 18:47:53.255520 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":295,"threads_started":1}
I20260619 18:47:53.258399 11797 raft_consensus.cc:1261] T 3d2edde41e714ca9842d98a4b891384f P 653a69448f244a2a99faad646667dda7 [term 1 FOLLOWER]: Refusing update from remote peer a236998d0ba646d693900cf720e416a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:47:53.258432 11936 raft_consensus.cc:1261] T 3d2edde41e714ca9842d98a4b891384f P 9ceae0288d68442b9b808ed536256768 [term 1 FOLLOWER]: Refusing update from remote peer a236998d0ba646d693900cf720e416a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:47:53.258950 12158 consensus_queue.cc:1035] T 3d2edde41e714ca9842d98a4b891384f P a236998d0ba646d693900cf720e416a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "653a69448f244a2a99faad646667dda7" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 35771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:47:53.259233 12226 consensus_queue.cc:1035] T 3d2edde41e714ca9842d98a4b891384f P a236998d0ba646d693900cf720e416a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ceae0288d68442b9b808ed536256768" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 45059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:47:53.279660 11640 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42514:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:47:53.279893 11640 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42514:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:47:53.282558 11640 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 022c3c6be95b46f8bf8e526a33780f6d: Sending DeleteTablet for 3 replicas of tablet 3d2edde41e714ca9842d98a4b891384f
I20260619 18:47:53.283336 12055 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d2edde41e714ca9842d98a4b891384f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:47:53 UTC) from {username='oryx', principal='oryx/127.3.243.126@KRBTEST.COM'} at 127.0.0.1:33140
I20260619 18:47:53.283561 11776 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d2edde41e714ca9842d98a4b891384f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:47:53 UTC) from {username='oryx', principal='oryx/127.3.243.126@KRBTEST.COM'} at 127.0.0.1:39282
I20260619 18:47:53.283644 12209 tablet_replica.cc:330] T 3d2edde41e714ca9842d98a4b891384f P a236998d0ba646d693900cf720e416a1: stopping tablet replica
I20260619 18:47:53.283689 11916 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d2edde41e714ca9842d98a4b891384f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:47:53 UTC) from {username='oryx', principal='oryx/127.3.243.126@KRBTEST.COM'} at 127.0.0.1:38872
I20260619 18:47:53.283857 12209 raft_consensus.cc:2229] T 3d2edde41e714ca9842d98a4b891384f P a236998d0ba646d693900cf720e416a1 [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:47:53.283962 12174 tablet_replica.cc:330] T 3d2edde41e714ca9842d98a4b891384f P 9ceae0288d68442b9b808ed536256768: stopping tablet replica
I20260619 18:47:53.284117 12174 raft_consensus.cc:2229] T 3d2edde41e714ca9842d98a4b891384f P 9ceae0288d68442b9b808ed536256768 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:47:53.284190 12209 raft_consensus.cc:2258] T 3d2edde41e714ca9842d98a4b891384f P a236998d0ba646d693900cf720e416a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:47:53.284356 12210 tablet_replica.cc:330] T 3d2edde41e714ca9842d98a4b891384f P 653a69448f244a2a99faad646667dda7: stopping tablet replica
I20260619 18:47:53.284375 12174 raft_consensus.cc:2258] T 3d2edde41e714ca9842d98a4b891384f P 9ceae0288d68442b9b808ed536256768 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:47:53.284575 12210 raft_consensus.cc:2229] T 3d2edde41e714ca9842d98a4b891384f P 653a69448f244a2a99faad646667dda7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:47:53.284762 12210 raft_consensus.cc:2258] T 3d2edde41e714ca9842d98a4b891384f P 653a69448f244a2a99faad646667dda7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:47:53.285436 12209 ts_tablet_manager.cc:1905] T 3d2edde41e714ca9842d98a4b891384f P a236998d0ba646d693900cf720e416a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:47:53.285984 12174 ts_tablet_manager.cc:1905] T 3d2edde41e714ca9842d98a4b891384f P 9ceae0288d68442b9b808ed536256768: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:47:53.289105 12210 ts_tablet_manager.cc:1905] T 3d2edde41e714ca9842d98a4b891384f P 653a69448f244a2a99faad646667dda7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:47:53.290916 12174 ts_tablet_manager.cc:1918] T 3d2edde41e714ca9842d98a4b891384f P 9ceae0288d68442b9b808ed536256768: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:47:53.291013 12174 log.cc:1196] T 3d2edde41e714ca9842d98a4b891384f P 9ceae0288d68442b9b808ed536256768: Deleting WAL directory at /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/minicluster-data/ts-1/wal/wals/3d2edde41e714ca9842d98a4b891384f
I20260619 18:47:53.291317 12174 ts_tablet_manager.cc:1939] T 3d2edde41e714ca9842d98a4b891384f P 9ceae0288d68442b9b808ed536256768: Deleting consensus metadata
I20260619 18:47:53.292223 11627 catalog_manager.cc:4841] TS 9ceae0288d68442b9b808ed536256768 (127.3.243.66:45059): tablet 3d2edde41e714ca9842d98a4b891384f (table test-table [id=3773263a0ffa4c3c9e7137278f645f8a]) successfully deleted
I20260619 18:47:53.292248 11640 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42514:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:47:53.293022 11640 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:47:53.299471 11776 tablet_service.cc:1467] Processing CreateTablet for tablet f6d4dbb9f3fc4022a921aad6513fe5ce (DEFAULT_TABLE table=test-table [id=f5f3c1cf53ec4e9b85c04da84c66bb54]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:47:53.299772 11916 tablet_service.cc:1467] Processing CreateTablet for tablet f6d4dbb9f3fc4022a921aad6513fe5ce (DEFAULT_TABLE table=test-table [id=f5f3c1cf53ec4e9b85c04da84c66bb54]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:47:53.299907 11776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6d4dbb9f3fc4022a921aad6513fe5ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:47:53.300182 11916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6d4dbb9f3fc4022a921aad6513fe5ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:47:53.303661 12209 ts_tablet_manager.cc:1918] T 3d2edde41e714ca9842d98a4b891384f P a236998d0ba646d693900cf720e416a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:47:53.304198 12210 ts_tablet_manager.cc:1918] T 3d2edde41e714ca9842d98a4b891384f P 653a69448f244a2a99faad646667dda7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:47:53.304404 12209 log.cc:1196] T 3d2edde41e714ca9842d98a4b891384f P a236998d0ba646d693900cf720e416a1: Deleting WAL directory at /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/minicluster-data/ts-2/wal/wals/3d2edde41e714ca9842d98a4b891384f
I20260619 18:47:53.304800 12055 tablet_service.cc:1467] Processing CreateTablet for tablet f6d4dbb9f3fc4022a921aad6513fe5ce (DEFAULT_TABLE table=test-table [id=f5f3c1cf53ec4e9b85c04da84c66bb54]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:47:53.305997 12055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6d4dbb9f3fc4022a921aad6513fe5ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:47:53.304587 12210 log.cc:1196] T 3d2edde41e714ca9842d98a4b891384f P 653a69448f244a2a99faad646667dda7: Deleting WAL directory at /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/minicluster-data/ts-0/wal/wals/3d2edde41e714ca9842d98a4b891384f
I20260619 18:47:53.306468 12210 ts_tablet_manager.cc:1939] T 3d2edde41e714ca9842d98a4b891384f P 653a69448f244a2a99faad646667dda7: Deleting consensus metadata
I20260619 18:47:53.306996 12152 tablet_bootstrap.cc:492] T f6d4dbb9f3fc4022a921aad6513fe5ce P 9ceae0288d68442b9b808ed536256768: Bootstrap starting.
I20260619 18:47:53.307411 11625 catalog_manager.cc:4841] TS 653a69448f244a2a99faad646667dda7 (127.3.243.65:35771): tablet 3d2edde41e714ca9842d98a4b891384f (table test-table [id=3773263a0ffa4c3c9e7137278f645f8a]) successfully deleted
I20260619 18:47:53.308521 12152 tablet_bootstrap.cc:654] T f6d4dbb9f3fc4022a921aad6513fe5ce P 9ceae0288d68442b9b808ed536256768: Neither blocks nor log segments found. Creating new log.
I20260619 18:47:53.310709 12152 tablet_bootstrap.cc:492] T f6d4dbb9f3fc4022a921aad6513fe5ce P 9ceae0288d68442b9b808ed536256768: No bootstrap required, opened a new log
I20260619 18:47:53.310869 12152 ts_tablet_manager.cc:1397] T f6d4dbb9f3fc4022a921aad6513fe5ce P 9ceae0288d68442b9b808ed536256768: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260619 18:47:53.311499 12152 raft_consensus.cc:348] T f6d4dbb9f3fc4022a921aad6513fe5ce P 9ceae0288d68442b9b808ed536256768 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653a69448f244a2a99faad646667dda7" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 35771 } } peers { permanent_uuid: "9ceae0288d68442b9b808ed536256768" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 45059 } } peers { permanent_uuid: "a236998d0ba646d693900cf720e416a1" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40189 } }
I20260619 18:47:53.311698 12152 raft_consensus.cc:374] T f6d4dbb9f3fc4022a921aad6513fe5ce P 9ceae0288d68442b9b808ed536256768 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:47:53.311775 12152 raft_consensus.cc:729] T f6d4dbb9f3fc4022a921aad6513fe5ce P 9ceae0288d68442b9b808ed536256768 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ceae0288d68442b9b808ed536256768, State: Initialized, Role: FOLLOWER
I20260619 18:47:53.312052 12152 consensus_queue.cc:260] T f6d4dbb9f3fc4022a921aad6513fe5ce P 9ceae0288d68442b9b808ed536256768 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653a69448f244a2a99faad646667dda7" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 35771 } } peers { permanent_uuid: "9ceae0288d68442b9b808ed536256768" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 45059 } } peers { permanent_uuid: "a236998d0ba646d693900cf720e416a1" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40189 } }
I20260619 18:47:53.312845 12160 raft_consensus.cc:482] T f6d4dbb9f3fc4022a921aad6513fe5ce P 9ceae0288d68442b9b808ed536256768 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:47:53.313038 12160 raft_consensus.cc:504] T f6d4dbb9f3fc4022a921aad6513fe5ce P 9ceae0288d68442b9b808ed536256768 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653a69448f244a2a99faad646667dda7" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 35771 } } peers { permanent_uuid: "9ceae0288d68442b9b808ed536256768" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 45059 } } peers { permanent_uuid: "a236998d0ba646d693900cf720e416a1" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40189 } }
I20260619 18:47:53.314618 12151 tablet_bootstrap.cc:492] T f6d4dbb9f3fc4022a921aad6513fe5ce P a236998d0ba646d693900cf720e416a1: Bootstrap starting.
I20260619 18:47:53.314982 12160 leader_election.cc:288] T f6d4dbb9f3fc4022a921aad6513fe5ce P 9ceae0288d68442b9b808ed536256768 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 653a69448f244a2a99faad646667dda7 (127.3.243.65:35771), a236998d0ba646d693900cf720e416a1 (127.3.243.67:40189)
I20260619 18:47:53.316430 12151 tablet_bootstrap.cc:654] T f6d4dbb9f3fc4022a921aad6513fe5ce P a236998d0ba646d693900cf720e416a1: Neither blocks nor log segments found. Creating new log.
I20260619 18:47:53.318945 12150 tablet_bootstrap.cc:492] T f6d4dbb9f3fc4022a921aad6513fe5ce P 653a69448f244a2a99faad646667dda7: Bootstrap starting.
I20260619 18:47:53.320794 12150 tablet_bootstrap.cc:654] T f6d4dbb9f3fc4022a921aad6513fe5ce P 653a69448f244a2a99faad646667dda7: Neither blocks nor log segments found. Creating new log.
I20260619 18:47:53.306010 12209 ts_tablet_manager.cc:1939] T 3d2edde41e714ca9842d98a4b891384f P a236998d0ba646d693900cf720e416a1: Deleting consensus metadata
I20260619 18:47:53.332562 11625 catalog_manager.cc:4841] TS a236998d0ba646d693900cf720e416a1 (127.3.243.67:40189): tablet 3d2edde41e714ca9842d98a4b891384f (table test-table [id=3773263a0ffa4c3c9e7137278f645f8a]) successfully deleted
I20260619 18:47:53.338618 12150 tablet_bootstrap.cc:492] T f6d4dbb9f3fc4022a921aad6513fe5ce P 653a69448f244a2a99faad646667dda7: No bootstrap required, opened a new log
I20260619 18:47:53.338959 12150 ts_tablet_manager.cc:1397] T f6d4dbb9f3fc4022a921aad6513fe5ce P 653a69448f244a2a99faad646667dda7: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.005s
I20260619 18:47:53.339865 12150 raft_consensus.cc:348] T f6d4dbb9f3fc4022a921aad6513fe5ce P 653a69448f244a2a99faad646667dda7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653a69448f244a2a99faad646667dda7" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 35771 } } peers { permanent_uuid: "9ceae0288d68442b9b808ed536256768" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 45059 } } peers { permanent_uuid: "a236998d0ba646d693900cf720e416a1" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40189 } }
I20260619 18:47:53.340587 12150 raft_consensus.cc:374] T f6d4dbb9f3fc4022a921aad6513fe5ce P 653a69448f244a2a99faad646667dda7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:47:53.340734 12150 raft_consensus.cc:729] T f6d4dbb9f3fc4022a921aad6513fe5ce P 653a69448f244a2a99faad646667dda7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 653a69448f244a2a99faad646667dda7, State: Initialized, Role: FOLLOWER
I20260619 18:47:53.341104 12150 consensus_queue.cc:260] T f6d4dbb9f3fc4022a921aad6513fe5ce P 653a69448f244a2a99faad646667dda7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653a69448f244a2a99faad646667dda7" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 35771 } } peers { permanent_uuid: "9ceae0288d68442b9b808ed536256768" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 45059 } } peers { permanent_uuid: "a236998d0ba646d693900cf720e416a1" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40189 } }
I20260619 18:47:53.342835 12150 ts_tablet_manager.cc:1428] T f6d4dbb9f3fc4022a921aad6513fe5ce P 653a69448f244a2a99faad646667dda7: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260619 18:47:53.344012 12151 tablet_bootstrap.cc:492] T f6d4dbb9f3fc4022a921aad6513fe5ce P a236998d0ba646d693900cf720e416a1: No bootstrap required, opened a new log
I20260619 18:47:53.344239 12151 ts_tablet_manager.cc:1397] T f6d4dbb9f3fc4022a921aad6513fe5ce P a236998d0ba646d693900cf720e416a1: Time spent bootstrapping tablet: real 0.030s user 0.005s sys 0.006s
I20260619 18:47:53.345103 12151 raft_consensus.cc:348] T f6d4dbb9f3fc4022a921aad6513fe5ce P a236998d0ba646d693900cf720e416a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653a69448f244a2a99faad646667dda7" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 35771 } } peers { permanent_uuid: "9ceae0288d68442b9b808ed536256768" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 45059 } } peers { permanent_uuid: "a236998d0ba646d693900cf720e416a1" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40189 } }
I20260619 18:47:53.345443 12151 raft_consensus.cc:374] T f6d4dbb9f3fc4022a921aad6513fe5ce P a236998d0ba646d693900cf720e416a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:47:53.345572 12151 raft_consensus.cc:729] T f6d4dbb9f3fc4022a921aad6513fe5ce P a236998d0ba646d693900cf720e416a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a236998d0ba646d693900cf720e416a1, State: Initialized, Role: FOLLOWER
I20260619 18:47:53.345849 12151 consensus_queue.cc:260] T f6d4dbb9f3fc4022a921aad6513fe5ce P a236998d0ba646d693900cf720e416a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653a69448f244a2a99faad646667dda7" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 35771 } } peers { permanent_uuid: "9ceae0288d68442b9b808ed536256768" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 45059 } } peers { permanent_uuid: "a236998d0ba646d693900cf720e416a1" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40189 } }
I20260619 18:47:53.346417 12151 ts_tablet_manager.cc:1428] T f6d4dbb9f3fc4022a921aad6513fe5ce P a236998d0ba646d693900cf720e416a1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 18:47:53.347322 12152 ts_tablet_manager.cc:1428] T f6d4dbb9f3fc4022a921aad6513fe5ce P 9ceae0288d68442b9b808ed536256768: Time spent starting tablet: real 0.036s user 0.012s sys 0.023s
I20260619 18:47:53.349862 12197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:53.317202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.66:43397 (local address 127.3.243.65:35771)
0619 18:47:53.317428 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:53.317434 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:53.330639 (+ 13205us) server_negotiation.cc:407] Connection header received
0619 18:47:53.331978 (+ 1339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:53.331984 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:53.332062 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:53.332124 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:47:53.334486 (+ 2362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:53.335743 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:53.346343 (+ 10600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:53.347794 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:53.347929 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:53.348034 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:53.349460 (+ 1426us) server_negotiation.cc:299] Negotiation successful
0619 18:47:53.349603 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 18:47:53.350580 12234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:53.316836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.65:35771 (local address 127.3.243.66:43397)
0619 18:47:53.330526 (+ 13690us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:53.330549 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:53.331683 (+ 1134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:53.332383 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:53.332388 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:53.332415 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:47:53.332714 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:53.332725 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:53.335996 (+ 3271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:53.336002 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:53.338431 (+ 2429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:53.338456 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:53.348190 (+ 9734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:53.348195 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:53.348593 (+ 398us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:53.348628 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 18:47:53.350117 (+ 1489us) client_negotiation.cc:240] Negotiation successful
0619 18:47:53.350265 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13524,"thread_start_us":13820,"threads_started":1}
I20260619 18:47:53.351203 11797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6d4dbb9f3fc4022a921aad6513fe5ce" candidate_uuid: "9ceae0288d68442b9b808ed536256768" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "653a69448f244a2a99faad646667dda7" is_pre_election: true
I20260619 18:47:53.351429 11797 raft_consensus.cc:2454] T f6d4dbb9f3fc4022a921aad6513fe5ce P 653a69448f244a2a99faad646667dda7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ceae0288d68442b9b808ed536256768 in term 0.
I20260619 18:47:53.351943 11868 leader_election.cc:302] T f6d4dbb9f3fc4022a921aad6513fe5ce P 9ceae0288d68442b9b808ed536256768 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 653a69448f244a2a99faad646667dda7, 9ceae0288d68442b9b808ed536256768; no voters:
I20260619 18:47:53.352440 12233 raft_consensus.cc:2781] T f6d4dbb9f3fc4022a921aad6513fe5ce P 9ceae0288d68442b9b808ed536256768 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:47:53.352622 12233 raft_consensus.cc:482] T f6d4dbb9f3fc4022a921aad6513fe5ce P 9ceae0288d68442b9b808ed536256768 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:47:53.352802 12233 raft_consensus.cc:3037] T f6d4dbb9f3fc4022a921aad6513fe5ce P 9ceae0288d68442b9b808ed536256768 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:47:53.356428 12229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:53.331139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.66:39255 (local address 127.3.243.67:40189)
0619 18:47:53.331306 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:53.331313 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:53.341561 (+ 10248us) server_negotiation.cc:407] Connection header received
0619 18:47:53.348776 (+ 7215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:53.348781 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:53.348847 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:53.348904 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:47:53.351117 (+ 2213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:53.352218 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:53.354609 (+ 2391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:53.355506 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:53.355650 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:53.355722 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:53.356032 (+ 310us) server_negotiation.cc:299] Negotiation successful
0619 18:47:53.356191 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 18:47:53.356485 12237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:53.331471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.67:40189 (local address 127.3.243.66:39255)
0619 18:47:53.341460 (+ 9989us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:53.341481 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:53.348459 (+ 6978us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:53.349078 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:53.349082 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:53.349107 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:47:53.350931 (+ 1824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:53.350941 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:53.352599 (+ 1658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:53.352604 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:53.354480 (+ 1876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:53.354492 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:53.355738 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:53.355745 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:53.355882 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:53.355905 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 18:47:53.355996 (+ 91us) client_negotiation.cc:240] Negotiation successful
0619 18:47:53.356131 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9837,"thread_start_us":10109,"threads_started":1}
I20260619 18:47:53.356411 12233 raft_consensus.cc:504] T f6d4dbb9f3fc4022a921aad6513fe5ce P 9ceae0288d68442b9b808ed536256768 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653a69448f244a2a99faad646667dda7" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 35771 } } peers { permanent_uuid: "9ceae0288d68442b9b808ed536256768" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 45059 } } peers { permanent_uuid: "a236998d0ba646d693900cf720e416a1" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40189 } }
I20260619 18:47:53.357190 12233 leader_election.cc:288] T f6d4dbb9f3fc4022a921aad6513fe5ce P 9ceae0288d68442b9b808ed536256768 [CANDIDATE]: Term 1 election: Requested vote from peers 653a69448f244a2a99faad646667dda7 (127.3.243.65:35771), a236998d0ba646d693900cf720e416a1 (127.3.243.67:40189)
I20260619 18:47:53.357244 12075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6d4dbb9f3fc4022a921aad6513fe5ce" candidate_uuid: "9ceae0288d68442b9b808ed536256768" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a236998d0ba646d693900cf720e416a1" is_pre_election: true
I20260619 18:47:53.357493 12075 raft_consensus.cc:2454] T f6d4dbb9f3fc4022a921aad6513fe5ce P a236998d0ba646d693900cf720e416a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ceae0288d68442b9b808ed536256768 in term 0.
I20260619 18:47:53.357570 11797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6d4dbb9f3fc4022a921aad6513fe5ce" candidate_uuid: "9ceae0288d68442b9b808ed536256768" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "653a69448f244a2a99faad646667dda7"
I20260619 18:47:53.357795 11797 raft_consensus.cc:3037] T f6d4dbb9f3fc4022a921aad6513fe5ce P 653a69448f244a2a99faad646667dda7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:47:53.359963 11797 raft_consensus.cc:2454] T f6d4dbb9f3fc4022a921aad6513fe5ce P 653a69448f244a2a99faad646667dda7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ceae0288d68442b9b808ed536256768 in term 1.
I20260619 18:47:53.360522 11868 leader_election.cc:302] T f6d4dbb9f3fc4022a921aad6513fe5ce P 9ceae0288d68442b9b808ed536256768 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 653a69448f244a2a99faad646667dda7, 9ceae0288d68442b9b808ed536256768; no voters:
I20260619 18:47:53.360844 12075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6d4dbb9f3fc4022a921aad6513fe5ce" candidate_uuid: "9ceae0288d68442b9b808ed536256768" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a236998d0ba646d693900cf720e416a1"
I20260619 18:47:53.360901 12233 raft_consensus.cc:2781] T f6d4dbb9f3fc4022a921aad6513fe5ce P 9ceae0288d68442b9b808ed536256768 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:47:53.361042 12075 raft_consensus.cc:3037] T f6d4dbb9f3fc4022a921aad6513fe5ce P a236998d0ba646d693900cf720e416a1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:47:53.361090 12233 raft_consensus.cc:686] T f6d4dbb9f3fc4022a921aad6513fe5ce P 9ceae0288d68442b9b808ed536256768 [term 1 LEADER]: Becoming Leader. State: Replica: 9ceae0288d68442b9b808ed536256768, State: Running, Role: LEADER
I20260619 18:47:53.361526 12233 consensus_queue.cc:237] T f6d4dbb9f3fc4022a921aad6513fe5ce P 9ceae0288d68442b9b808ed536256768 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653a69448f244a2a99faad646667dda7" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 35771 } } peers { permanent_uuid: "9ceae0288d68442b9b808ed536256768" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 45059 } } peers { permanent_uuid: "a236998d0ba646d693900cf720e416a1" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40189 } }
I20260619 18:47:53.363322 12075 raft_consensus.cc:2454] T f6d4dbb9f3fc4022a921aad6513fe5ce P a236998d0ba646d693900cf720e416a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ceae0288d68442b9b808ed536256768 in term 1.
I20260619 18:47:53.365502 11639 catalog_manager.cc:5495] T f6d4dbb9f3fc4022a921aad6513fe5ce P 9ceae0288d68442b9b808ed536256768 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ceae0288d68442b9b808ed536256768 (127.3.243.66). New cstate: current_term: 1 leader_uuid: "9ceae0288d68442b9b808ed536256768" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653a69448f244a2a99faad646667dda7" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 35771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ceae0288d68442b9b808ed536256768" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 45059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a236998d0ba646d693900cf720e416a1" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40189 } health_report { overall_health: UNKNOWN } } }
I20260619 18:47:53.401355 12240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:53.395014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38884 (local address 127.3.243.66:45059)
0619 18:47:53.395581 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:53.395586 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:53.395613 (+ 27us) server_negotiation.cc:407] Connection header received
0619 18:47:53.395715 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:53.395720 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:53.395820 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:53.396011 (+ 191us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:47:53.396367 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:53.397543 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:53.398741 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:53.399391 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:53.399513 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:53.399899 (+ 386us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:47:53.400660 (+ 761us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:47:53.400789 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:53.400878 (+ 89us) server_negotiation.cc:299] Negotiation successful
0619 18:47:53.400964 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":298,"threads_started":1}
I20260619 18:47:53.417331 12197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:53.411026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.66:52583 (local address 127.3.243.65:35771)
0619 18:47:53.411237 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:53.411244 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:53.411739 (+ 495us) server_negotiation.cc:407] Connection header received
0619 18:47:53.411918 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:53.411922 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:53.412036 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:53.412213 (+ 177us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:47:53.412712 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:53.413649 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:53.415617 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:53.416097 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:53.416240 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:53.416702 (+ 462us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:47:53.416817 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:47:53.416878 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:53.417025 (+ 147us) server_negotiation.cc:299] Negotiation successful
0619 18:47:53.417098 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 18:47:53.417510 12243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:53.410993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.65:35771 (local address 127.3.243.66:52583)
0619 18:47:53.411635 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:53.411655 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:53.411797 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:53.412227 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:53.412233 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:53.412247 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 18:47:53.412578 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:53.412590 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:53.413781 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:53.413787 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:53.415488 (+ 1701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:53.415503 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:53.416307 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:53.416313 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:53.416480 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:53.416552 (+ 72us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 18:47:53.416905 (+ 353us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 18:47:53.416916 (+ 11us) client_negotiation.cc:769] Sending connection context
0619 18:47:53.417006 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 18:47:53.417140 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":306,"threads_started":1}
I20260619 18:47:53.425659 12075 raft_consensus.cc:1261] T f6d4dbb9f3fc4022a921aad6513fe5ce P a236998d0ba646d693900cf720e416a1 [term 1 FOLLOWER]: Refusing update from remote peer 9ceae0288d68442b9b808ed536256768: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:47:53.425809 11797 raft_consensus.cc:1261] T f6d4dbb9f3fc4022a921aad6513fe5ce P 653a69448f244a2a99faad646667dda7 [term 1 FOLLOWER]: Refusing update from remote peer 9ceae0288d68442b9b808ed536256768: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:47:53.426476 12233 consensus_queue.cc:1035] T f6d4dbb9f3fc4022a921aad6513fe5ce P 9ceae0288d68442b9b808ed536256768 [LEADER]: Connected to new peer: Peer: permanent_uuid: "653a69448f244a2a99faad646667dda7" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 35771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:47:53.426769 12160 consensus_queue.cc:1035] T f6d4dbb9f3fc4022a921aad6513fe5ce P 9ceae0288d68442b9b808ed536256768 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a236998d0ba646d693900cf720e416a1" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:47:53.453701 12240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:53.448289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.65:38561 (local address 127.3.243.66:45059)
0619 18:47:53.448457 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:53.448463 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:53.448866 (+ 403us) server_negotiation.cc:407] Connection header received
0619 18:47:53.448994 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:53.448999 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:53.449072 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:53.449162 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:47:53.449618 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:53.450646 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:53.451997 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:53.452464 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:53.452570 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:53.453004 (+ 434us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:47:53.453120 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:47:53.453193 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:53.453323 (+ 130us) server_negotiation.cc:299] Negotiation successful
0619 18:47:53.453395 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 18:47:53.453809 12248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:53.448232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.66:45059 (local address 127.3.243.65:38561)
0619 18:47:53.448778 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:53.448793 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:53.448890 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:53.449164 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:53.449167 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:53.449176 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 18:47:53.449462 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:53.449475 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:53.450762 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:53.450767 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:53.451869 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:53.451881 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:53.452644 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:53.452651 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:53.452804 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:53.452860 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 18:47:53.453213 (+ 353us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 18:47:53.453223 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 18:47:53.453296 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 18:47:53.453426 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":236,"threads_started":1}
I20260619 18:47:53.495834 11639 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42514:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:47:53.496177 11639 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42514:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:47:53.499627 11639 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 022c3c6be95b46f8bf8e526a33780f6d: Sending DeleteTablet for 3 replicas of tablet f6d4dbb9f3fc4022a921aad6513fe5ce
I20260619 18:47:53.500507 11916 tablet_service.cc:1514] Processing DeleteTablet for tablet f6d4dbb9f3fc4022a921aad6513fe5ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:47:53 UTC) from {username='oryx', principal='oryx/127.3.243.126@KRBTEST.COM'} at 127.0.0.1:38872
I20260619 18:47:53.500787 12174 tablet_replica.cc:330] T f6d4dbb9f3fc4022a921aad6513fe5ce P 9ceae0288d68442b9b808ed536256768: stopping tablet replica
I20260619 18:47:53.501006 12174 raft_consensus.cc:2229] T f6d4dbb9f3fc4022a921aad6513fe5ce P 9ceae0288d68442b9b808ed536256768 [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:47:53.501037 11776 tablet_service.cc:1514] Processing DeleteTablet for tablet f6d4dbb9f3fc4022a921aad6513fe5ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:47:53 UTC) from {username='oryx', principal='oryx/127.3.243.126@KRBTEST.COM'} at 127.0.0.1:39282
I20260619 18:47:53.501375 12210 tablet_replica.cc:330] T f6d4dbb9f3fc4022a921aad6513fe5ce P 653a69448f244a2a99faad646667dda7: stopping tablet replica
I20260619 18:47:53.501518 12174 raft_consensus.cc:2258] T f6d4dbb9f3fc4022a921aad6513fe5ce P 9ceae0288d68442b9b808ed536256768 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:47:53.501580 12210 raft_consensus.cc:2229] T f6d4dbb9f3fc4022a921aad6513fe5ce P 653a69448f244a2a99faad646667dda7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:47:53.501802 12210 raft_consensus.cc:2258] T f6d4dbb9f3fc4022a921aad6513fe5ce P 653a69448f244a2a99faad646667dda7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:47:53.501818 12055 tablet_service.cc:1514] Processing DeleteTablet for tablet f6d4dbb9f3fc4022a921aad6513fe5ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:47:53 UTC) from {username='oryx', principal='oryx/127.3.243.126@KRBTEST.COM'} at 127.0.0.1:33140
I20260619 18:47:53.502302 12209 tablet_replica.cc:330] T f6d4dbb9f3fc4022a921aad6513fe5ce P a236998d0ba646d693900cf720e416a1: stopping tablet replica
I20260619 18:47:53.502488 12209 raft_consensus.cc:2229] T f6d4dbb9f3fc4022a921aad6513fe5ce P a236998d0ba646d693900cf720e416a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:47:53.502689 12209 raft_consensus.cc:2258] T f6d4dbb9f3fc4022a921aad6513fe5ce P a236998d0ba646d693900cf720e416a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:47:53.503633 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 11711
I20260619 18:47:53.504299 12209 ts_tablet_manager.cc:1905] T f6d4dbb9f3fc4022a921aad6513fe5ce P a236998d0ba646d693900cf720e416a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:47:53.509368 12174 ts_tablet_manager.cc:1905] T f6d4dbb9f3fc4022a921aad6513fe5ce P 9ceae0288d68442b9b808ed536256768: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:47:53.513193 12210 ts_tablet_manager.cc:1905] T f6d4dbb9f3fc4022a921aad6513fe5ce P 653a69448f244a2a99faad646667dda7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:47:53.517966 12209 ts_tablet_manager.cc:1918] T f6d4dbb9f3fc4022a921aad6513fe5ce P a236998d0ba646d693900cf720e416a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 18:47:53.520001 12174 ts_tablet_manager.cc:1918] T f6d4dbb9f3fc4022a921aad6513fe5ce P 9ceae0288d68442b9b808ed536256768: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 18:47:53.520231 12209 log.cc:1196] T f6d4dbb9f3fc4022a921aad6513fe5ce P a236998d0ba646d693900cf720e416a1: Deleting WAL directory at /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/minicluster-data/ts-2/wal/wals/f6d4dbb9f3fc4022a921aad6513fe5ce
I20260619 18:47:53.520349 12210 ts_tablet_manager.cc:1918] T f6d4dbb9f3fc4022a921aad6513fe5ce P 653a69448f244a2a99faad646667dda7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 18:47:53.520581 12210 log.cc:1196] T f6d4dbb9f3fc4022a921aad6513fe5ce P 653a69448f244a2a99faad646667dda7: Deleting WAL directory at /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/minicluster-data/ts-0/wal/wals/f6d4dbb9f3fc4022a921aad6513fe5ce
I20260619 18:47:53.520732 12209 ts_tablet_manager.cc:1939] T f6d4dbb9f3fc4022a921aad6513fe5ce P a236998d0ba646d693900cf720e416a1: Deleting consensus metadata
I20260619 18:47:53.521183 12210 ts_tablet_manager.cc:1939] T f6d4dbb9f3fc4022a921aad6513fe5ce P 653a69448f244a2a99faad646667dda7: Deleting consensus metadata
I20260619 18:47:53.521781 12174 log.cc:1196] T f6d4dbb9f3fc4022a921aad6513fe5ce P 9ceae0288d68442b9b808ed536256768: Deleting WAL directory at /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894810715282-4045-0/minicluster-data/ts-1/wal/wals/f6d4dbb9f3fc4022a921aad6513fe5ce
I20260619 18:47:53.521903 11625 catalog_manager.cc:4841] TS a236998d0ba646d693900cf720e416a1 (127.3.243.67:40189): tablet f6d4dbb9f3fc4022a921aad6513fe5ce (table test-table [id=f5f3c1cf53ec4e9b85c04da84c66bb54]) successfully deleted
I20260619 18:47:53.522677 12174 ts_tablet_manager.cc:1939] T f6d4dbb9f3fc4022a921aad6513fe5ce P 9ceae0288d68442b9b808ed536256768: Deleting consensus metadata
I20260619 18:47:53.523160 11625 catalog_manager.cc:4841] TS 653a69448f244a2a99faad646667dda7 (127.3.243.65:35771): tablet f6d4dbb9f3fc4022a921aad6513fe5ce (table test-table [id=f5f3c1cf53ec4e9b85c04da84c66bb54]) successfully deleted
I20260619 18:47:53.523813 11627 catalog_manager.cc:4841] TS 9ceae0288d68442b9b808ed536256768 (127.3.243.66:45059): tablet f6d4dbb9f3fc4022a921aad6513fe5ce (table test-table [id=f5f3c1cf53ec4e9b85c04da84c66bb54]) successfully deleted
Jun 19 18:47:53 dist-test-slave-m9wg krb5kdc[11595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894872, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.243.65@KRBTEST.COM
I20260619 18:47:53.542948 12197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:53.505261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39308 (local address 127.3.243.65:35771)
0619 18:47:53.505428 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:53.505434 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:53.524748 (+ 19314us) server_negotiation.cc:407] Connection header received
0619 18:47:53.524937 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:53.524943 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:53.525060 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:53.525170 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:53.526701 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:53.527792 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:53.529631 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:53.530404 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:53.530544 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:53.535618 (+ 5074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:53.535636 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:53.535657 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:53.535704 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:53.539666 (+ 3962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:53.540524 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:53.540534 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:53.540537 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:53.540705 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:53.541128 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:53.541134 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:53.541137 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:53.541689 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:53.542009 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:53.542340 (+ 331us) server_negotiation.cc:299] Negotiation successful
0619 18:47:53.542607 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 18:47:53.544034 11842 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:47:53.747673 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 11711
I20260619 18:47:53.768030 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 11851
Jun 19 18:47:53 dist-test-slave-m9wg krb5kdc[11595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894872, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.243.66@KRBTEST.COM
I20260619 18:47:53.785180 12240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:53.768781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38896 (local address 127.3.243.66:45059)
0619 18:47:53.768969 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:53.768973 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:53.768992 (+ 19us) server_negotiation.cc:407] Connection header received
0619 18:47:53.769213 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:53.769217 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:53.769289 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:53.769373 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:53.771038 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:53.771873 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:53.773686 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:53.774140 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:53.774238 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:53.779230 (+ 4992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:53.779241 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:53.779255 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:53.779286 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:53.782517 (+ 3231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:53.783302 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:53.783308 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:53.783311 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:53.783431 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:53.783788 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:53.783791 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:53.783793 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:53.784124 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:53.784404 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:53.784677 (+ 273us) server_negotiation.cc:299] Negotiation successful
0619 18:47:53.784842 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 18:47:53.785853 11981 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:47:53.913049 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 11851
I20260619 18:47:53.929664 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 11990
Jun 19 18:47:53 dist-test-slave-m9wg krb5kdc[11595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894872, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.243.67@KRBTEST.COM
I20260619 18:47:53.943562 12254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:53.930323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33164 (local address 127.3.243.67:40189)
0619 18:47:53.930766 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:53.930770 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:53.930784 (+ 14us) server_negotiation.cc:407] Connection header received
0619 18:47:53.930834 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:53.930838 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:53.930893 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:53.930982 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:53.932224 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:53.932926 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:53.934109 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:53.934481 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:53.934601 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:53.938705 (+ 4104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:53.938719 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:53.938738 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:53.938770 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:53.941438 (+ 2668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:53.941969 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:53.941975 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:53.941977 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:53.942086 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:53.942359 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:53.942362 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:53.942364 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:53.942703 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:53.942932 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:53.943088 (+ 156us) server_negotiation.cc:299] Negotiation successful
0619 18:47:53.943275 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":212,"threads_started":1}
I20260619 18:47:53.944124 12120 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:47:54.062059 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 11990
I20260619 18:47:54.079030 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 11608
I20260619 18:47:54.080175 11680 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:47:54.193543 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 11608
2026-06-19T18:47:54Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6369 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781894810715282-4045-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:47:54 dist-test-slave-m9wg krb5kdc[12259](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:47:54 dist-test-slave-m9wg krb5kdc[12259](info): set up 2 sockets
Jun 19 18:47:54 dist-test-slave-m9wg krb5kdc[12259](info): commencing operation
krb5kdc: starting...
W20260619 18:47:55.200793 4045 mini_kdc.cc:121] Time spent starting KDC: real 0.968s user 0.002s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781894810715282-4045-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781894810715282-4045-0/krb5kdc/test-user.keytab.
Jun 19 18:47:55 dist-test-slave-m9wg krb5kdc[12259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894875, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:47:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:47:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.243.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab.
Entry for principal kudu/127.3.243.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab.
I20260619 18:47:55.383317 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781894810715282-4045-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781894810715282-4045-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781894810715282-4045-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.126:46335
--webserver_interface=127.3.243.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab
--principal=kudu/127.3.243.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:43365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.126:46335
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:47:55.681792 12272 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:47:55.682101 12272 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:47:55.682222 12272 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:47:55.689213 12272 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:47:55.689327 12272 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:47:55.689406 12272 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:47:55.689483 12272 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:47:55.689548 12272 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:47:55.699139 12272 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:43365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781894810715282-4045-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781894810715282-4045-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.243.126:46335
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.126:46335
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.243.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781894810715282-4045-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:47:55.700423 12272 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:47:55.701668 12272 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:47:55.712455 12280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:55.712407 12278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:55.712496 12277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:47:55.713272 12272 server_base.cc:1034] running on GCE node
I20260619 18:47:55.714008 12272 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:47:55.715318 12272 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:47:55.716513 12272 hybrid_clock.cc:648] HybridClock initialized: now 1781894875716471 us; error 31 us; skew 500 ppm
I20260619 18:47:55.716828 12272 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-19T18:47:55Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1594 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:47:55 dist-test-slave-m9wg krb5kdc[12286](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:47:55 dist-test-slave-m9wg krb5kdc[12286](info): set up 2 sockets
Jun 19 18:47:55 dist-test-slave-m9wg krb5kdc[12286](info): commencing operation
krb5kdc: starting...
W20260619 18:47:56.780860 4045 mini_kdc.cc:121] Time spent starting KDC: real 0.953s user 0.002s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/krb5kdc/test-user.keytab.
Jun 19 18:47:56 dist-test-slave-m9wg krb5kdc[12286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894876, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:47:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:47:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.243.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab.
Entry for principal kudu/127.3.243.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab.
I20260619 18:47:56.965253 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.126:46457
--webserver_interface=127.3.243.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab
--principal=kudu/127.3.243.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:33069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.126:46457
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:47:57.263026 12299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:47:57.263299 12299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:47:57.263428 12299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:47:57.270321 12299 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:47:57.270430 12299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:47:57.270507 12299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:47:57.270581 12299 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:47:57.270637 12299 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:47:57.280036 12299 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:33069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.243.126:46457
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab
--principal=kudu/127.3.243.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.126:46457
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.243.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:47:57.281271 12299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:47:57.282557 12299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:47:57.296500 12305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:57.296464 12304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:57.296532 12307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:47:57.297240 12299 server_base.cc:1034] running on GCE node
I20260619 18:47:57.298027 12299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:47:57.299376 12299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:47:57.300593 12299 hybrid_clock.cc:648] HybridClock initialized: now 1781894877300535 us; error 39 us; skew 500 ppm
I20260619 18:47:57.300908 12299 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:47:57 dist-test-slave-m9wg krb5kdc[12286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894877, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:47:57.303815 12299 init.cc:377] Logged in from keytab as kudu/127.3.243.126@KRBTEST.COM (short username kudu)
I20260619 18:47:57.305227 12299 webserver.cc:460] Webserver started at http://127.3.243.126:35075/ using document root <none> and password file <none>
I20260619 18:47:57.305857 12299 fs_manager.cc:359] Metadata directory not provided
I20260619 18:47:57.305966 12299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:47:57.306245 12299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:47:57.309085 12299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/master-0/data/instance:
uuid: "141329533b94426d91076014bc32bbac"
format_stamp: "Formatted at 2026-06-19 18:47:57 on dist-test-slave-m9wg"
server_key: "8b2c91500a0ecb18b724d2ae31ddac92"
server_key_iv: "73c90eca815051e93640c9b0a8beaaea"
server_key_version: "encryptionkey@0"
I20260619 18:47:57.309847 12299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/master-0/wal/instance:
uuid: "141329533b94426d91076014bc32bbac"
format_stamp: "Formatted at 2026-06-19 18:47:57 on dist-test-slave-m9wg"
server_key: "8b2c91500a0ecb18b724d2ae31ddac92"
server_key_iv: "73c90eca815051e93640c9b0a8beaaea"
server_key_version: "encryptionkey@0"
I20260619 18:47:57.314488 12299 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 18:47:57.317699 12314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:57.318923 12299 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 18:47:57.319070 12299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/master-0/wal
uuid: "141329533b94426d91076014bc32bbac"
format_stamp: "Formatted at 2026-06-19 18:47:57 on dist-test-slave-m9wg"
server_key: "8b2c91500a0ecb18b724d2ae31ddac92"
server_key_iv: "73c90eca815051e93640c9b0a8beaaea"
server_key_version: "encryptionkey@0"
I20260619 18:47:57.319235 12299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:47:57.335335 12299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:47:57.339371 12299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:47:57.339628 12299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:47:57.356554 12299 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.126:46457
I20260619 18:47:57.356575 12365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.126:46457 every 8 connection(s)
I20260619 18:47:57.357872 12299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
I20260619 18:47:57.360582 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 12299
I20260619 18:47:57.360761 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/master-0/wal/instance
I20260619 18:47:57.361373 4045 external_mini_cluster.cc:1428] Setting key a106bb7a2024e1329d0ef8841bf786b8
I20260619 18:47:57.361578 12366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:47:57.375051 12366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 141329533b94426d91076014bc32bbac: Bootstrap starting.
Jun 19 18:47:57 dist-test-slave-m9wg krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894876, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.126@KRBTEST.COM
I20260619 18:47:57.378696 12366 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 141329533b94426d91076014bc32bbac: Neither blocks nor log segments found. Creating new log.
I20260619 18:47:57.379978 12366 log.cc:824] T 00000000000000000000000000000000 P 141329533b94426d91076014bc32bbac: Log is configured to *not* fsync() on all Append() calls
I20260619 18:47:57.383132 12366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 141329533b94426d91076014bc32bbac: No bootstrap required, opened a new log
I20260619 18:47:57.384294 12369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:57.363196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56958 (local address 127.3.243.126:46457)
0619 18:47:57.363984 (+ 788us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:57.364003 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:57.364049 (+ 46us) server_negotiation.cc:407] Connection header received
0619 18:47:57.366190 (+ 2141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:57.366217 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:57.366723 (+ 506us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:57.367146 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:57.368442 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:57.370331 (+ 1889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:57.371515 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:57.372591 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:57.372771 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:57.376573 (+ 3802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:57.376584 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:57.376602 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:57.376646 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:57.380207 (+ 3561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:57.380918 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:57.380926 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:57.380934 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:57.381102 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:57.381472 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:57.381476 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:57.381479 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:57.382100 (+ 621us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:57.382362 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:57.382642 (+ 280us) server_negotiation.cc:299] Negotiation successful
0619 18:47:57.383026 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":325,"threads_started":1}
I20260619 18:47:57.390992 12366 raft_consensus.cc:348] T 00000000000000000000000000000000 P 141329533b94426d91076014bc32bbac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "141329533b94426d91076014bc32bbac" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 46457 } }
I20260619 18:47:57.391304 12366 raft_consensus.cc:374] T 00000000000000000000000000000000 P 141329533b94426d91076014bc32bbac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:47:57.391371 12366 raft_consensus.cc:729] T 00000000000000000000000000000000 P 141329533b94426d91076014bc32bbac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 141329533b94426d91076014bc32bbac, State: Initialized, Role: FOLLOWER
I20260619 18:47:57.391868 12366 consensus_queue.cc:260] T 00000000000000000000000000000000 P 141329533b94426d91076014bc32bbac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "141329533b94426d91076014bc32bbac" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 46457 } }
I20260619 18:47:57.392078 12366 raft_consensus.cc:388] T 00000000000000000000000000000000 P 141329533b94426d91076014bc32bbac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:47:57.392203 12366 raft_consensus.cc:482] T 00000000000000000000000000000000 P 141329533b94426d91076014bc32bbac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:47:57.392330 12366 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 141329533b94426d91076014bc32bbac [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:47:57.394287 12366 raft_consensus.cc:504] T 00000000000000000000000000000000 P 141329533b94426d91076014bc32bbac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "141329533b94426d91076014bc32bbac" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 46457 } }
I20260619 18:47:57.394691 12366 leader_election.cc:302] T 00000000000000000000000000000000 P 141329533b94426d91076014bc32bbac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 141329533b94426d91076014bc32bbac; no voters:
I20260619 18:47:57.395200 12366 leader_election.cc:288] T 00000000000000000000000000000000 P 141329533b94426d91076014bc32bbac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:47:57.395330 12371 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 141329533b94426d91076014bc32bbac [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:47:57.395941 12371 raft_consensus.cc:686] T 00000000000000000000000000000000 P 141329533b94426d91076014bc32bbac [term 1 LEADER]: Becoming Leader. State: Replica: 141329533b94426d91076014bc32bbac, State: Running, Role: LEADER
I20260619 18:47:57.396562 12371 consensus_queue.cc:237] T 00000000000000000000000000000000 P 141329533b94426d91076014bc32bbac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "141329533b94426d91076014bc32bbac" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 46457 } }
I20260619 18:47:57.397296 12366 sys_catalog.cc:564] T 00000000000000000000000000000000 P 141329533b94426d91076014bc32bbac [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:47:57.399089 12372 sys_catalog.cc:455] T 00000000000000000000000000000000 P 141329533b94426d91076014bc32bbac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "141329533b94426d91076014bc32bbac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "141329533b94426d91076014bc32bbac" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 46457 } } }
I20260619 18:47:57.399111 12373 sys_catalog.cc:455] T 00000000000000000000000000000000 P 141329533b94426d91076014bc32bbac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 141329533b94426d91076014bc32bbac. Latest consensus state: current_term: 1 leader_uuid: "141329533b94426d91076014bc32bbac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "141329533b94426d91076014bc32bbac" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 46457 } } }
I20260619 18:47:57.399483 12372 sys_catalog.cc:458] T 00000000000000000000000000000000 P 141329533b94426d91076014bc32bbac [sys.catalog]: This master's current role is: LEADER
I20260619 18:47:57.400172 12373 sys_catalog.cc:458] T 00000000000000000000000000000000 P 141329533b94426d91076014bc32bbac [sys.catalog]: This master's current role is: LEADER
I20260619 18:47:57.401577 12377 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:47:57.406589 12377 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:47:57.415100 12377 catalog_manager.cc:1347] Generated new cluster ID: 067fbde9aae24d489a564505dc7f717d
I20260619 18:47:57.415197 12377 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:47:57.431258 12377 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:47:57.432473 12377 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:47:57.441921 12377 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 141329533b94426d91076014bc32bbac: Generated new TSK 0
I20260619 18:47:57.442651 12377 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.243.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.65.keytab.
Entry for principal kudu/127.3.243.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.65.keytab.
I20260619 18:47:57.499188 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.65:0
--local_ip_for_outbound_sockets=127.3.243.65
--webserver_interface=127.3.243.65
--webserver_port=0
--tserver_master_addrs=127.3.243.126:46457
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.65.keytab
--principal=kudu/127.3.243.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:33069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:47:57.826856 12392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:47:57.827121 12392 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:47:57.827255 12392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:47:57.834252 12392 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:47:57.834358 12392 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:47:57.834502 12392 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.65
I20260619 18:47:57.843840 12392 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:33069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.65.keytab
--principal=kudu/127.3.243.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.243.65
--webserver_port=0
--tserver_master_addrs=127.3.243.126:46457
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.65
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:47:57.845065 12392 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:47:57.846293 12392 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:47:57.858448 12398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:57.858522 12397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:57.858496 12400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:47:57.859325 12392 server_base.cc:1034] running on GCE node
I20260619 18:47:57.859863 12392 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:47:57.860752 12392 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:47:57.861987 12392 hybrid_clock.cc:648] HybridClock initialized: now 1781894877861931 us; error 39 us; skew 500 ppm
I20260619 18:47:57.862293 12392 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:47:57 dist-test-slave-m9wg krb5kdc[12286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894877, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:47:57.865286 12392 init.cc:377] Logged in from keytab as kudu/127.3.243.65@KRBTEST.COM (short username kudu)
I20260619 18:47:57.866551 12392 webserver.cc:460] Webserver started at http://127.3.243.65:34367/ using document root <none> and password file <none>
I20260619 18:47:57.867189 12392 fs_manager.cc:359] Metadata directory not provided
I20260619 18:47:57.867298 12392 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:47:57.867565 12392 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:47:57.870316 12392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/ts-0/data/instance:
uuid: "099264bb565943c58037283ba521ef5b"
format_stamp: "Formatted at 2026-06-19 18:47:57 on dist-test-slave-m9wg"
server_key: "77cc5d3208fa934b6d4112f63002b9ab"
server_key_iv: "486d516cdc5a949b78ed7f68275f2292"
server_key_version: "encryptionkey@0"
I20260619 18:47:57.871047 12392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/ts-0/wal/instance:
uuid: "099264bb565943c58037283ba521ef5b"
format_stamp: "Formatted at 2026-06-19 18:47:57 on dist-test-slave-m9wg"
server_key: "77cc5d3208fa934b6d4112f63002b9ab"
server_key_iv: "486d516cdc5a949b78ed7f68275f2292"
server_key_version: "encryptionkey@0"
I20260619 18:47:57.875702 12392 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260619 18:47:57.878877 12407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:57.880054 12392 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 18:47:57.880223 12392 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/ts-0/wal
uuid: "099264bb565943c58037283ba521ef5b"
format_stamp: "Formatted at 2026-06-19 18:47:57 on dist-test-slave-m9wg"
server_key: "77cc5d3208fa934b6d4112f63002b9ab"
server_key_iv: "486d516cdc5a949b78ed7f68275f2292"
server_key_version: "encryptionkey@0"
I20260619 18:47:57.880383 12392 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:47:57.908627 12392 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:47:57.912568 12392 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:47:57.912812 12392 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:47:57.913661 12392 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:47:57.914851 12392 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:47:57.914937 12392 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:57.915046 12392 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:47:57.915096 12392 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:57.943331 12392 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.65:41923
I20260619 18:47:57.943361 12519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.65:41923 every 8 connection(s)
I20260619 18:47:57.944542 12392 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/ts-0/data/info.pb
I20260619 18:47:57.949508 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 12392
I20260619 18:47:57.949697 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/ts-0/wal/instance
I20260619 18:47:57.950500 4045 external_mini_cluster.cc:1428] Setting key 5de6771822d0b961476b38dc1a289381
Jun 19 18:47:57 dist-test-slave-m9wg krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894877, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.65@KRBTEST.COM for kudu/127.3.243.126@KRBTEST.COM
I20260619 18:47:57.964037 12522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:57.946819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.65:33789 (local address 127.3.243.126:46457)
0619 18:47:57.947204 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:57.947209 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:57.948124 (+ 915us) server_negotiation.cc:407] Connection header received
0619 18:47:57.950247 (+ 2123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:57.950252 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:57.950319 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:57.950413 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:57.952143 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:57.952880 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:57.954996 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:57.955394 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:57.955482 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:57.958985 (+ 3503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:57.958989 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:57.958993 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:57.959023 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:57.961339 (+ 2316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:57.961970 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:57.961975 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:57.961977 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:57.962089 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:57.962464 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:57.962469 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:57.962472 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:57.962718 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:57.962842 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:57.963476 (+ 634us) server_negotiation.cc:299] Negotiation successful
0619 18:47:57.963697 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":182,"threads_started":1}
I20260619 18:47:57.964884 12523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:57.947329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:46457 (local address 127.3.243.65:33789)
0619 18:47:57.947961 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:57.948004 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:57.949967 (+ 1963us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:57.950671 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:57.950682 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:57.951293 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:47:57.951985 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:57.952009 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:57.953045 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:57.953050 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:57.954847 (+ 1797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:57.954863 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:57.955586 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:57.955592 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:57.955837 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:57.956523 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:47:57.956549 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:47:57.958779 (+ 2230us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:47:57.961462 (+ 2683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:57.961471 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:57.961486 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:57.961863 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:57.962188 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:57.962193 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:57.962197 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:57.962362 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:57.962834 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:47:57.962841 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:47:57.963134 (+ 293us) client_negotiation.cc:769] Sending connection context
0619 18:47:57.963361 (+ 227us) client_negotiation.cc:240] Negotiation successful
0619 18:47:57.963821 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":240,"threads_started":1}
I20260619 18:47:57.966264 12520 heartbeater.cc:347] Connected to a master server at 127.3.243.126:46457
I20260619 18:47:57.966579 12520 heartbeater.cc:464] Registering TS with master...
I20260619 18:47:57.967399 12520 heartbeater.cc:511] Master 127.3.243.126:46457 requested a full tablet report, sending...
I20260619 18:47:57.969769 12331 ts_manager.cc:194] Registered new tserver with Master: 099264bb565943c58037283ba521ef5b (127.3.243.65:41923)
WARNING: no policy specified for kudu/127.3.243.66@KRBTEST.COM; defaulting to no policy
I20260619 18:47:57.971489 12331 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.65@KRBTEST.COM'} at 127.3.243.65:33789
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.66.keytab.
Entry for principal kudu/127.3.243.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.66.keytab.
I20260619 18:47:57.994509 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.66:0
--local_ip_for_outbound_sockets=127.3.243.66
--webserver_interface=127.3.243.66
--webserver_port=0
--tserver_master_addrs=127.3.243.126:46457
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.66.keytab
--principal=kudu/127.3.243.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:33069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:47:58.294090 12526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:47:58.294411 12526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:47:58.294567 12526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:47:58.301685 12526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:47:58.301779 12526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:47:58.301926 12526 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.66
I20260619 18:47:58.311072 12526 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:33069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.66.keytab
--principal=kudu/127.3.243.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.243.66
--webserver_port=0
--tserver_master_addrs=127.3.243.126:46457
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.66
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:47:58.312222 12526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:47:58.313419 12526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:47:58.325278 12531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:58.325325 12532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:58.325358 12534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:47:58.326092 12526 server_base.cc:1034] running on GCE node
I20260619 18:47:58.326617 12526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:47:58.327478 12526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:47:58.328665 12526 hybrid_clock.cc:648] HybridClock initialized: now 1781894878328607 us; error 46 us; skew 500 ppm
I20260619 18:47:58.328940 12526 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:47:58 dist-test-slave-m9wg krb5kdc[12286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894878, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:47:58.331678 12526 init.cc:377] Logged in from keytab as kudu/127.3.243.66@KRBTEST.COM (short username kudu)
I20260619 18:47:58.332971 12526 webserver.cc:460] Webserver started at http://127.3.243.66:44203/ using document root <none> and password file <none>
I20260619 18:47:58.333571 12526 fs_manager.cc:359] Metadata directory not provided
I20260619 18:47:58.333683 12526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:47:58.333956 12526 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:47:58.336644 12526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/ts-1/data/instance:
uuid: "810defb7cc7f4b7a93732c941dedbaa1"
format_stamp: "Formatted at 2026-06-19 18:47:58 on dist-test-slave-m9wg"
server_key: "405299098167c7caeb7c84e57f63ef4f"
server_key_iv: "544389ad389bad044baa01430ada9f0f"
server_key_version: "encryptionkey@0"
I20260619 18:47:58.337349 12526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/ts-1/wal/instance:
uuid: "810defb7cc7f4b7a93732c941dedbaa1"
format_stamp: "Formatted at 2026-06-19 18:47:58 on dist-test-slave-m9wg"
server_key: "405299098167c7caeb7c84e57f63ef4f"
server_key_iv: "544389ad389bad044baa01430ada9f0f"
server_key_version: "encryptionkey@0"
I20260619 18:47:58.341944 12526 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 18:47:58.345082 12541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:58.346254 12526 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 18:47:58.346391 12526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/ts-1/wal
uuid: "810defb7cc7f4b7a93732c941dedbaa1"
format_stamp: "Formatted at 2026-06-19 18:47:58 on dist-test-slave-m9wg"
server_key: "405299098167c7caeb7c84e57f63ef4f"
server_key_iv: "544389ad389bad044baa01430ada9f0f"
server_key_version: "encryptionkey@0"
I20260619 18:47:58.346568 12526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:47:58.379642 12526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:47:58.383148 12526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:47:58.383383 12526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:47:58.384303 12526 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:47:58.385594 12526 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:47:58.385663 12526 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:58.385776 12526 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:47:58.385823 12526 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:58.414199 12526 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.66:46659
I20260619 18:47:58.414240 12653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.66:46659 every 8 connection(s)
I20260619 18:47:58.415343 12526 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/ts-1/data/info.pb
I20260619 18:47:58.420405 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 12526
I20260619 18:47:58.420555 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/ts-1/wal/instance
I20260619 18:47:58.421075 4045 external_mini_cluster.cc:1428] Setting key 6a78b323ab4dede0c156aecf5549c565
Jun 19 18:47:58 dist-test-slave-m9wg krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894878, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.66@KRBTEST.COM for kudu/127.3.243.126@KRBTEST.COM
I20260619 18:47:58.435105 12522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:58.417688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.66:56887 (local address 127.3.243.126:46457)
0619 18:47:58.417855 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:58.417860 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:58.418889 (+ 1029us) server_negotiation.cc:407] Connection header received
0619 18:47:58.420663 (+ 1774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:58.420670 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:58.420770 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:58.420878 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:58.422456 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:58.423315 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:58.425491 (+ 2176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:58.425905 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:58.425999 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:58.429829 (+ 3830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:58.429835 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:58.429839 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:58.429876 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:58.432371 (+ 2495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:58.433111 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:58.433116 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:58.433118 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:58.433239 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:58.433659 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:58.433662 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:58.433664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:58.433901 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:58.434004 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:58.434711 (+ 707us) server_negotiation.cc:299] Negotiation successful
0619 18:47:58.434861 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 18:47:58.436393 12656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:58.418103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:46457 (local address 127.3.243.66:56887)
0619 18:47:58.418738 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:58.418779 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:58.420386 (+ 1607us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:58.421057 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:58.421064 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:58.421591 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:47:58.422308 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:58.422334 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:58.423519 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:58.423524 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:58.425363 (+ 1839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:58.425379 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:58.426110 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:58.426117 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:58.426353 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:58.427000 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:47:58.427031 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:47:58.429591 (+ 2560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:47:58.432490 (+ 2899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:58.432498 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:58.432512 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:58.433011 (+ 499us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:58.433331 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:58.433335 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:58.433338 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:58.433564 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:58.434009 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:47:58.434016 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:47:58.434392 (+ 376us) client_negotiation.cc:769] Sending connection context
0619 18:47:58.434656 (+ 264us) client_negotiation.cc:240] Negotiation successful
0619 18:47:58.435098 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":258,"threads_started":1}
I20260619 18:47:58.437808 12654 heartbeater.cc:347] Connected to a master server at 127.3.243.126:46457
I20260619 18:47:58.438091 12654 heartbeater.cc:464] Registering TS with master...
I20260619 18:47:58.438721 12654 heartbeater.cc:511] Master 127.3.243.126:46457 requested a full tablet report, sending...
I20260619 18:47:58.440176 12331 ts_manager.cc:194] Registered new tserver with Master: 810defb7cc7f4b7a93732c941dedbaa1 (127.3.243.66:46659)
I20260619 18:47:58.440970 12331 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.66@KRBTEST.COM'} at 127.3.243.66:56887
WARNING: no policy specified for kudu/127.3.243.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.67.keytab.
Entry for principal kudu/127.3.243.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.67.keytab.
I20260619 18:47:58.465046 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.67:0
--local_ip_for_outbound_sockets=127.3.243.67
--webserver_interface=127.3.243.67
--webserver_port=0
--tserver_master_addrs=127.3.243.126:46457
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.67.keytab
--principal=kudu/127.3.243.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:33069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:47:58.762087 12659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:47:58.762362 12659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:47:58.762485 12659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:47:58.769405 12659 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:47:58.769517 12659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:47:58.769666 12659 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.67
I20260619 18:47:58.778815 12659 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:33069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.67.keytab
--principal=kudu/127.3.243.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.243.67
--webserver_port=0
--tserver_master_addrs=127.3.243.126:46457
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.67
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:47:58.779963 12659 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:47:58.781227 12659 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:47:58.793233 12664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:58.793205 12665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:58.793876 12667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:47:58.794200 12659 server_base.cc:1034] running on GCE node
I20260619 18:47:58.794736 12659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:47:58.795578 12659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:47:58.796763 12659 hybrid_clock.cc:648] HybridClock initialized: now 1781894878796722 us; error 29 us; skew 500 ppm
I20260619 18:47:58.797075 12659 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:47:58 dist-test-slave-m9wg krb5kdc[12286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894878, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:47:58.799975 12659 init.cc:377] Logged in from keytab as kudu/127.3.243.67@KRBTEST.COM (short username kudu)
I20260619 18:47:58.801357 12659 webserver.cc:460] Webserver started at http://127.3.243.67:34499/ using document root <none> and password file <none>
I20260619 18:47:58.802014 12659 fs_manager.cc:359] Metadata directory not provided
I20260619 18:47:58.802124 12659 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:47:58.802398 12659 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:47:58.805145 12659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/ts-2/data/instance:
uuid: "ea3429273de043a78478a6b20aa08cfd"
format_stamp: "Formatted at 2026-06-19 18:47:58 on dist-test-slave-m9wg"
server_key: "12a5ec8f94a09c4be599e22c14c910f1"
server_key_iv: "cf60ac92c128bc05be38f338f8ce62e8"
server_key_version: "encryptionkey@0"
I20260619 18:47:58.805822 12659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/ts-2/wal/instance:
uuid: "ea3429273de043a78478a6b20aa08cfd"
format_stamp: "Formatted at 2026-06-19 18:47:58 on dist-test-slave-m9wg"
server_key: "12a5ec8f94a09c4be599e22c14c910f1"
server_key_iv: "cf60ac92c128bc05be38f338f8ce62e8"
server_key_version: "encryptionkey@0"
I20260619 18:47:58.810418 12659 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 18:47:58.813623 12674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:58.814810 12659 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 18:47:58.814949 12659 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/ts-2/wal
uuid: "ea3429273de043a78478a6b20aa08cfd"
format_stamp: "Formatted at 2026-06-19 18:47:58 on dist-test-slave-m9wg"
server_key: "12a5ec8f94a09c4be599e22c14c910f1"
server_key_iv: "cf60ac92c128bc05be38f338f8ce62e8"
server_key_version: "encryptionkey@0"
I20260619 18:47:58.815111 12659 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:47:58.834158 12659 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:47:58.838147 12659 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:47:58.838394 12659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:47:58.839324 12659 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:47:58.840610 12659 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:47:58.840679 12659 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:58.840792 12659 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:47:58.840838 12659 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:58.869590 12659 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.67:40123
I20260619 18:47:58.869633 12786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.67:40123 every 8 connection(s)
I20260619 18:47:58.870764 12659 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/ts-2/data/info.pb
I20260619 18:47:58.871137 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 12659
I20260619 18:47:58.871281 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/ts-2/wal/instance
I20260619 18:47:58.871852 4045 external_mini_cluster.cc:1428] Setting key 388fc6a5be8ab661cfb3c8063ee33adb
Jun 19 18:47:58 dist-test-slave-m9wg krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894878, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.67@KRBTEST.COM for kudu/127.3.243.126@KRBTEST.COM
I20260619 18:47:58.889518 12522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:58.873244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.67:42443 (local address 127.3.243.126:46457)
0619 18:47:58.873403 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:58.873407 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:58.874768 (+ 1361us) server_negotiation.cc:407] Connection header received
0619 18:47:58.876753 (+ 1985us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:58.876757 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:58.876824 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:58.876894 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:58.878260 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:58.878984 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:58.880504 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:58.880909 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:58.881009 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:58.884143 (+ 3134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:58.884170 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:58.884175 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:58.884213 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:58.887051 (+ 2838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:58.887643 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:58.887646 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:58.887648 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:58.887755 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:58.888064 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:58.888067 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:58.888069 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:58.888354 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:58.888456 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:58.889109 (+ 653us) server_negotiation.cc:299] Negotiation successful
0619 18:47:58.889272 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 18:47:58.890380 12789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:58.873845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:46457 (local address 127.3.243.67:42443)
0619 18:47:58.874618 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0619 18:47:58.874668 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 18:47:58.876517 (+ 1849us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:47:58.877078 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:47:58.877085 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:47:58.877577 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:47:58.878111 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:58.878128 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:58.879102 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:58.879105 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:58.880347 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:47:58.880358 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:47:58.881117 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:47:58.881123 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:47:58.881317 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:58.881888 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:47:58.881914 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:47:58.883942 (+ 2028us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:47:58.887154 (+ 3212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:58.887160 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:58.887171 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:58.887562 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:58.887825 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:47:58.887828 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:47:58.887830 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:47:58.887990 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:47:58.888468 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:47:58.888473 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:47:58.888754 (+ 281us) client_negotiation.cc:769] Sending connection context
0619 18:47:58.888988 (+ 234us) client_negotiation.cc:240] Negotiation successful
0619 18:47:58.889367 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":298,"threads_started":1}
I20260619 18:47:58.891654 12787 heartbeater.cc:347] Connected to a master server at 127.3.243.126:46457
I20260619 18:47:58.891935 12787 heartbeater.cc:464] Registering TS with master...
I20260619 18:47:58.892614 12787 heartbeater.cc:511] Master 127.3.243.126:46457 requested a full tablet report, sending...
I20260619 18:47:58.893898 12331 ts_manager.cc:194] Registered new tserver with Master: ea3429273de043a78478a6b20aa08cfd (127.3.243.67:40123)
I20260619 18:47:58.894640 12331 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.67@KRBTEST.COM'} at 127.3.243.67:42443
I20260619 18:47:58.898977 4045 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.3.243.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.243.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.243.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/krb5kdc/oryx_127.3.243.65.keytab.
Entry for principal oryx/127.3.243.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/krb5kdc/oryx_127.3.243.65.keytab.
WARNING: no policy specified for oryx/127.3.243.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.243.66@KRBTEST.COM" created.
I20260619 18:47:58.974356 12520 heartbeater.cc:503] Master 127.3.243.126:46457 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.243.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/krb5kdc/oryx_127.3.243.66.keytab.
Entry for principal oryx/127.3.243.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/krb5kdc/oryx_127.3.243.66.keytab.
WARNING: no policy specified for oryx/127.3.243.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.243.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.243.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/krb5kdc/oryx_127.3.243.67.keytab.
Entry for principal oryx/127.3.243.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/krb5kdc/oryx_127.3.243.67.keytab.
I20260619 18:47:59.016605 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 12392
Jun 19 18:47:59 dist-test-slave-m9wg krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894876, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.65@KRBTEST.COM
I20260619 18:47:59.031800 12797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:59.017699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46828 (local address 127.3.243.65:41923)
0619 18:47:59.018323 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:59.018333 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:59.018368 (+ 35us) server_negotiation.cc:407] Connection header received
0619 18:47:59.018444 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:59.018451 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:59.018659 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:59.018841 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:59.020223 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:59.021208 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:59.022477 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:59.023083 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:59.023193 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:59.027071 (+ 3878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:59.027081 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:59.027094 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:59.027126 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:59.029645 (+ 2519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:59.030221 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:59.030228 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:59.030231 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:59.030347 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:59.030630 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:59.030633 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:59.030635 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:59.030948 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:59.031155 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:59.031345 (+ 190us) server_negotiation.cc:299] Negotiation successful
0619 18:47:59.031524 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":284,"threads_started":1}
I20260619 18:47:59.033049 12515 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:47:59.112706 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 12392
I20260619 18:47:59.126019 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 12526
Jun 19 18:47:59 dist-test-slave-m9wg krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894876, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.66@KRBTEST.COM
I20260619 18:47:59.140576 12799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:59.126930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45476 (local address 127.3.243.66:46659)
0619 18:47:59.127427 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:59.127449 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:59.127473 (+ 24us) server_negotiation.cc:407] Connection header received
0619 18:47:59.127537 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:59.127543 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:59.127746 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:59.127868 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:59.129046 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:59.130053 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:59.131259 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:59.131804 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:59.131897 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:59.135678 (+ 3781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:59.135689 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:59.135703 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:59.135736 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:59.138256 (+ 2520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:59.138822 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:59.138829 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:59.138831 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:59.138944 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:59.139202 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:59.139205 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:59.139207 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:59.139524 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:59.139732 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:59.139984 (+ 252us) server_negotiation.cc:299] Negotiation successful
0619 18:47:59.140221 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":242,"threads_started":1}
I20260619 18:47:59.141744 12649 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:47:59.221737 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 12526
I20260619 18:47:59.234346 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 12659
Jun 19 18:47:59 dist-test-slave-m9wg krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894876, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.67@KRBTEST.COM
I20260619 18:47:59.249380 12801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:47:59.235213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45982 (local address 127.3.243.67:40123)
0619 18:47:59.235780 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0619 18:47:59.235787 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:47:59.235814 (+ 27us) server_negotiation.cc:407] Connection header received
0619 18:47:59.235884 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:47:59.235891 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:47:59.236099 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:47:59.236271 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:47:59.237535 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:59.238565 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:59.239795 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:47:59.240384 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:47:59.240497 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:47:59.244564 (+ 4067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:47:59.244574 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:47:59.244587 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:47:59.244617 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:47:59.247072 (+ 2455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:59.247635 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:59.247641 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:59.247644 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:59.247749 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:47:59.248016 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:47:59.248019 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:47:59.248021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:47:59.248465 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:47:59.248695 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0619 18:47:59.248891 (+ 196us) server_negotiation.cc:299] Negotiation successful
0619 18:47:59.249094 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":261,"threads_started":1}
I20260619 18:47:59.250591 12782 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:47:59.332093 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 12659
I20260619 18:47:59.344391 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 12299
I20260619 18:47:59.345633 12361 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:47:59.446727 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 12299
I20260619 18:47:59.463017 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.126:46457
--webserver_interface=127.3.243.126
--webserver_port=35075
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab
--principal=kudu/127.3.243.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:33069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.126:46457
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:47:59.766261 12804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:47:59.766536 12804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:47:59.766656 12804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:47:59.773607 12804 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:47:59.773702 12804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:47:59.773756 12804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:47:59.773833 12804 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:47:59.773881 12804 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:47:59.783537 12804 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:33069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.243.126:46457
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab
--principal=kudu/127.3.243.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.126:46457
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.243.126
--webserver_port=35075
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:47:59.784864 12804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:47:59.786145 12804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:47:59.797775 12810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:59.798015 12812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:47:59.798290 12809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:47:59.798978 12804 server_base.cc:1034] running on GCE node
I20260619 18:47:59.799912 12804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:47:59.801676 12804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:47:59.802901 12804 hybrid_clock.cc:648] HybridClock initialized: now 1781894879802840 us; error 43 us; skew 500 ppm
I20260619 18:47:59.803215 12804 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:47:59 dist-test-slave-m9wg krb5kdc[12286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894879, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:47:59.806066 12804 init.cc:377] Logged in from keytab as kudu/127.3.243.126@KRBTEST.COM (short username kudu)
I20260619 18:47:59.807418 12804 webserver.cc:460] Webserver started at http://127.3.243.126:35075/ using document root <none> and password file <none>
I20260619 18:47:59.808058 12804 fs_manager.cc:359] Metadata directory not provided
I20260619 18:47:59.808225 12804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:47:59.813171 12804 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260619 18:47:59.815665 12819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:47:59.816895 12804 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 18:47:59.817031 12804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/master-0/wal
uuid: "141329533b94426d91076014bc32bbac"
format_stamp: "Formatted at 2026-06-19 18:47:57 on dist-test-slave-m9wg"
server_key: "8b2c91500a0ecb18b724d2ae31ddac92"
server_key_iv: "73c90eca815051e93640c9b0a8beaaea"
server_key_version: "encryptionkey@0"
I20260619 18:47:59.817900 12804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:47:59.835290 12804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:47:59.839215 12804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:47:59.839448 12804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:47:59.855250 12804 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.126:46457
I20260619 18:47:59.855270 12870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.126:46457 every 8 connection(s)
I20260619 18:47:59.856681 12804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
I20260619 18:47:59.860780 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 12804
I20260619 18:47:59.861703 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.65:41923
--local_ip_for_outbound_sockets=127.3.243.65
--tserver_master_addrs=127.3.243.126:46457
--webserver_port=34367
--webserver_interface=127.3.243.65
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.65.keytab
--principal=kudu/127.3.243.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:33069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.3.243.65
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/krb5kdc/oryx_127.3.243.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 18:47:59.864318 12871 sys_catalog.cc:263] Verifying existing consensus state
I20260619 18:47:59.868669 12871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 141329533b94426d91076014bc32bbac: Bootstrap starting.
I20260619 18:47:59.886927 12871 log.cc:824] T 00000000000000000000000000000000 P 141329533b94426d91076014bc32bbac: Log is configured to *not* fsync() on all Append() calls
I20260619 18:47:59.895334 12871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 141329533b94426d91076014bc32bbac: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 18:47:59.895788 12871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 141329533b94426d91076014bc32bbac: Bootstrap complete.
I20260619 18:47:59.904352 12871 raft_consensus.cc:348] T 00000000000000000000000000000000 P 141329533b94426d91076014bc32bbac [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "141329533b94426d91076014bc32bbac" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 46457 } }
I20260619 18:47:59.905138 12871 raft_consensus.cc:729] T 00000000000000000000000000000000 P 141329533b94426d91076014bc32bbac [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 141329533b94426d91076014bc32bbac, State: Initialized, Role: FOLLOWER
I20260619 18:47:59.905727 12871 consensus_queue.cc:260] T 00000000000000000000000000000000 P 141329533b94426d91076014bc32bbac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "141329533b94426d91076014bc32bbac" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 46457 } }
I20260619 18:47:59.905930 12871 raft_consensus.cc:388] T 00000000000000000000000000000000 P 141329533b94426d91076014bc32bbac [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:47:59.906040 12871 raft_consensus.cc:482] T 00000000000000000000000000000000 P 141329533b94426d91076014bc32bbac [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:47:59.906184 12871 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 141329533b94426d91076014bc32bbac [term 1 FOLLOWER]: Advancing to term 2
I20260619 18:47:59.909724 12871 raft_consensus.cc:504] T 00000000000000000000000000000000 P 141329533b94426d91076014bc32bbac [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "141329533b94426d91076014bc32bbac" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 46457 } }
I20260619 18:47:59.910167 12871 leader_election.cc:302] T 00000000000000000000000000000000 P 141329533b94426d91076014bc32bbac [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 141329533b94426d91076014bc32bbac; no voters:
I20260619 18:47:59.910717 12871 leader_election.cc:288] T 00000000000000000000000000000000 P 141329533b94426d91076014bc32bbac [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 18:47:59.910848 12875 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 141329533b94426d91076014bc32bbac [term 2 FOLLOWER]: Leader election won for term 2
I20260619 18:47:59.911877 12875 raft_consensus.cc:686] T 00000000000000000000000000000000 P 141329533b94426d91076014bc32bbac [term 2 LEADER]: Becoming Leader. State: Replica: 141329533b94426d91076014bc32bbac, State: Running, Role: LEADER
I20260619 18:47:59.912483 12875 consensus_queue.cc:237] T 00000000000000000000000000000000 P 141329533b94426d91076014bc32bbac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "141329533b94426d91076014bc32bbac" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 46457 } }
I20260619 18:47:59.912765 12871 sys_catalog.cc:564] T 00000000000000000000000000000000 P 141329533b94426d91076014bc32bbac [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:47:59.914819 12877 sys_catalog.cc:455] T 00000000000000000000000000000000 P 141329533b94426d91076014bc32bbac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 141329533b94426d91076014bc32bbac. Latest consensus state: current_term: 2 leader_uuid: "141329533b94426d91076014bc32bbac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "141329533b94426d91076014bc32bbac" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 46457 } } }
I20260619 18:47:59.915145 12877 sys_catalog.cc:458] T 00000000000000000000000000000000 P 141329533b94426d91076014bc32bbac [sys.catalog]: This master's current role is: LEADER
I20260619 18:47:59.915536 12876 sys_catalog.cc:455] T 00000000000000000000000000000000 P 141329533b94426d91076014bc32bbac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "141329533b94426d91076014bc32bbac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "141329533b94426d91076014bc32bbac" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 46457 } } }
I20260619 18:47:59.915820 12876 sys_catalog.cc:458] T 00000000000000000000000000000000 P 141329533b94426d91076014bc32bbac [sys.catalog]: This master's current role is: LEADER
I20260619 18:47:59.926494 12881 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:47:59.931818 12881 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:47:59.933214 12881 catalog_manager.cc:1259] Loaded cluster ID: 067fbde9aae24d489a564505dc7f717d
I20260619 18:47:59.933367 12881 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:47:59.937206 12881 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:47:59.938589 12881 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 141329533b94426d91076014bc32bbac: Loaded TSK: 0
I20260619 18:47:59.940059 12881 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 18:48:00.177446 12873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:48:00.177699 12873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:48:00.177807 12873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:48:00.184820 12873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:48:00.184916 12873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:48:00.185045 12873 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.65
I20260619 18:48:00.194394 12873 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:33069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/krb5kdc/oryx_127.3.243.65.keytab
--principal=oryx/127.3.243.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.65:41923
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.243.65
--webserver_port=34367
--tserver_master_addrs=127.3.243.126:46457
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.65
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:48:00.195623 12873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:48:00.196879 12873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:48:00.210346 12897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:48:00.210436 12898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:48:00.211270 12900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:48:00.211788 12873 server_base.cc:1034] running on GCE node
I20260619 18:48:00.212368 12873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:48:00.213282 12873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:48:00.214469 12873 hybrid_clock.cc:648] HybridClock initialized: now 1781894880214411 us; error 56 us; skew 500 ppm
I20260619 18:48:00.214735 12873 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:48:00 dist-test-slave-m9wg krb5kdc[12286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894880, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.243.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:48:00.217537 12873 init.cc:377] Logged in from keytab as oryx/127.3.243.65@KRBTEST.COM (short username oryx)
I20260619 18:48:00.218847 12873 webserver.cc:460] Webserver started at http://127.3.243.65:34367/ using document root <none> and password file <none>
I20260619 18:48:00.219447 12873 fs_manager.cc:359] Metadata directory not provided
I20260619 18:48:00.219555 12873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:48:00.224707 12873 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 18:48:00.227272 12907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:48:00.228611 12873 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:48:00.228757 12873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/ts-0/wal
uuid: "099264bb565943c58037283ba521ef5b"
format_stamp: "Formatted at 2026-06-19 18:47:57 on dist-test-slave-m9wg"
server_key: "77cc5d3208fa934b6d4112f63002b9ab"
server_key_iv: "486d516cdc5a949b78ed7f68275f2292"
server_key_version: "encryptionkey@0"
I20260619 18:48:00.229633 12873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:48:00.244980 12873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:48:00.249040 12873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:48:00.249292 12873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:48:00.250227 12873 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:48:00.251544 12873 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:48:00.251638 12873 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:48:00.251745 12873 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:48:00.251809 12873 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:48:00.280866 12873 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.65:41923
I20260619 18:48:00.280929 13019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.65:41923 every 8 connection(s)
I20260619 18:48:00.282253 12873 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/ts-0/data/info.pb
I20260619 18:48:00.283828 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 12873
I20260619 18:48:00.284570 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.66:46659
--local_ip_for_outbound_sockets=127.3.243.66
--tserver_master_addrs=127.3.243.126:46457
--webserver_port=44203
--webserver_interface=127.3.243.66
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.66.keytab
--principal=kudu/127.3.243.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:33069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.3.243.66
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/krb5kdc/oryx_127.3.243.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 18:48:00 dist-test-slave-m9wg krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.65@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:48:00 dist-test-slave-m9wg krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.65@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM, Server not found in Kerberos database
W20260619 18:48:00.593617 13024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:48:00.593861 13024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:48:00.593986 13024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:48:00.600953 13024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:48:00.601049 13024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:48:00.601197 13024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.66
I20260619 18:48:00.610774 13024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:33069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/krb5kdc/oryx_127.3.243.66.keytab
--principal=oryx/127.3.243.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.66:46659
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.243.66
--webserver_port=44203
--tserver_master_addrs=127.3.243.126:46457
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.66
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:48:00.611982 13024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:48:00.613284 13024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:48:00.627899 13031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:48:00.628031 13033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:48:00.632078 13030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:48:00.633490 13024 server_base.cc:1034] running on GCE node
I20260619 18:48:00.634255 13024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:48:00.637588 13024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:48:00.638803 13024 hybrid_clock.cc:648] HybridClock initialized: now 1781894880638732 us; error 65 us; skew 500 ppm
I20260619 18:48:00.639201 13024 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:48:00 dist-test-slave-m9wg krb5kdc[12286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894880, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.243.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:48:00.643373 13024 init.cc:377] Logged in from keytab as oryx/127.3.243.66@KRBTEST.COM (short username oryx)
I20260619 18:48:00.645124 13024 webserver.cc:460] Webserver started at http://127.3.243.66:44203/ using document root <none> and password file <none>
I20260619 18:48:00.645828 13024 fs_manager.cc:359] Metadata directory not provided
I20260619 18:48:00.645951 13024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:48:00.651265 13024 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260619 18:48:00.654114 13040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:48:00.655617 13024 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 18:48:00.655764 13024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/ts-1/wal
uuid: "810defb7cc7f4b7a93732c941dedbaa1"
format_stamp: "Formatted at 2026-06-19 18:47:58 on dist-test-slave-m9wg"
server_key: "405299098167c7caeb7c84e57f63ef4f"
server_key_iv: "544389ad389bad044baa01430ada9f0f"
server_key_version: "encryptionkey@0"
I20260619 18:48:00.656726 13024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:48:00.679317 13024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:48:00.683331 13024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:48:00.683578 13024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:48:00.684610 13024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:48:00.685936 13024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:48:00.686028 13024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:48:00.686136 13024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:48:00.686203 13024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:48:00.724749 13024 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.66:46659
I20260619 18:48:00.724802 13152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.66:46659 every 8 connection(s)
I20260619 18:48:00.726120 13024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/ts-1/data/info.pb
I20260619 18:48:00.735095 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 13024
I20260619 18:48:00.735906 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.67:40123
--local_ip_for_outbound_sockets=127.3.243.67
--tserver_master_addrs=127.3.243.126:46457
--webserver_port=34499
--webserver_interface=127.3.243.67
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.67.keytab
--principal=kudu/127.3.243.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:33069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.3.243.67
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/krb5kdc/oryx_127.3.243.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 18:48:00 dist-test-slave-m9wg krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.66@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:48:00 dist-test-slave-m9wg krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.66@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM, Server not found in Kerberos database
W20260619 18:48:01.052443 13157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:48:01.052685 13157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:48:01.052811 13157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:48:01.059600 13157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:48:01.059710 13157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:48:01.059845 13157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.67
I20260619 18:48:01.069010 13157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:33069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/krb5kdc/oryx_127.3.243.67.keytab
--principal=oryx/127.3.243.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.67:40123
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.243.67
--webserver_port=34499
--tserver_master_addrs=127.3.243.126:46457
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.67
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:48:01.070159 13157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:48:01.071375 13157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:48:01.085276 13163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:48:01.088542 13164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:48:01.097556 13166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:48:01.098053 13157 server_base.cc:1034] running on GCE node
I20260619 18:48:01.098903 13157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:48:01.100292 13157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:48:01.101550 13157 hybrid_clock.cc:648] HybridClock initialized: now 1781894881101477 us; error 63 us; skew 500 ppm
I20260619 18:48:01.102005 13157 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:48:01 dist-test-slave-m9wg krb5kdc[12286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894881, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.243.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:48:01.110625 13157 init.cc:377] Logged in from keytab as oryx/127.3.243.67@KRBTEST.COM (short username oryx)
I20260619 18:48:01.113073 13157 webserver.cc:460] Webserver started at http://127.3.243.67:34499/ using document root <none> and password file <none>
I20260619 18:48:01.114290 13157 fs_manager.cc:359] Metadata directory not provided
I20260619 18:48:01.114473 13157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:48:01.123035 13157 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.005s sys 0.000s
I20260619 18:48:01.126932 13173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:48:01.129060 13157 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 18:48:01.129249 13157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/ts-2/wal
uuid: "ea3429273de043a78478a6b20aa08cfd"
format_stamp: "Formatted at 2026-06-19 18:47:58 on dist-test-slave-m9wg"
server_key: "12a5ec8f94a09c4be599e22c14c910f1"
server_key_iv: "cf60ac92c128bc05be38f338f8ce62e8"
server_key_version: "encryptionkey@0"
I20260619 18:48:01.130721 13157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:48:01.168682 13157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:48:01.175011 13157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:48:01.175330 13157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:48:01.176707 13157 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:48:01.178618 13157 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:48:01.178742 13157 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:48:01.178853 13157 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:48:01.178932 13157 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:48:01.224583 13157 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.67:40123
I20260619 18:48:01.224648 13285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.67:40123 every 8 connection(s)
I20260619 18:48:01.226756 13157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894810715282-4045-0/minicluster-data/ts-2/data/info.pb
I20260619 18:48:01.235481 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 13157
Jun 19 18:48:01 dist-test-slave-m9wg krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.67@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:48:01 dist-test-slave-m9wg krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.67@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:48:01.316519 13289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:48:01.236562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42482 (local address 127.3.243.126:46457)
0619 18:48:01.268303 (+ 31741us) server_negotiation.cc:206] Beginning negotiation
0619 18:48:01.268310 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:48:01.277415 (+ 9105us) server_negotiation.cc:407] Connection header received
0619 18:48:01.278588 (+ 1173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:48:01.278604 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:48:01.278708 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:48:01.278844 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:48:01.286411 (+ 7567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:01.287736 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:01.289889 (+ 2153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:01.290580 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:01.290699 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:01.298585 (+ 7886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:48:01.298597 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:48:01.298620 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:48:01.298827 (+ 207us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:48:01.303870 (+ 5043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:48:01.308316 (+ 4446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:48:01.308329 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:48:01.308342 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:48:01.308944 (+ 602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:48:01.312265 (+ 3321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:48:01.312272 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:48:01.312276 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:48:01.314035 (+ 1759us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:48:01.314343 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0619 18:48:01.314940 (+ 597us) server_negotiation.cc:299] Negotiation successful
0619 18:48:01.315433 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31620,"thread_start_us":29279,"threads_started":1}
W20260619 18:48:01.470335 13022 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.175s user 0.000s sys 0.012s
W20260619 18:48:01.472113 13023 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:48:00.285159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.65:41791 (local address 127.3.243.126:46457)
0619 18:48:00.286817 (+ 1658us) server_negotiation.cc:206] Beginning negotiation
0619 18:48:00.286832 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 18:48:00.286881 (+ 49us) server_negotiation.cc:407] Connection header received
0619 18:48:00.288368 (+ 1487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:48:00.288382 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:48:00.288699 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:48:00.289026 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:48:00.290462 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:00.291703 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:00.293244 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:00.293854 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:00.294037 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:01.471751 (+1177714us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.243.65:41791: BlockingRecv error: recv got EOF from 127.3.243.65:41791 (error 108)
Metrics: {"server-negotiator.queue_time_us":1406,"thread_start_us":1165,"threads_started":1}
I20260619 18:48:01.472805 13022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:48:00.285180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:46457 (local address 127.3.243.65:41791)
0619 18:48:00.285944 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0619 18:48:00.285987 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 18:48:00.287802 (+ 1815us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:48:00.289187 (+ 1385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:48:00.289199 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:48:00.289764 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:48:00.290255 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:00.290272 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:00.291856 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:00.291860 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:00.293113 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:00.293125 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:00.294077 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:00.294084 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:00.294326 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:00.294946 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:48:00.294974 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:48:01.471390 (+1176416us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.243.126:46457: Server oryx/127.3.243.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":301,"threads_started":1}
W20260619 18:48:01.473943 13020 heartbeater.cc:644] Failed to heartbeat to 127.3.243.126:46457 (0 consecutive failures): Not authorized: Failed to ping master at 127.3.243.126:46457: Client connection negotiation failed: client connection to 127.3.243.126:46457: Server oryx/127.3.243.126@KRBTEST.COM not found in Kerberos database
Jun 19 18:48:01 dist-test-slave-m9wg krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.65@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:48:01 dist-test-slave-m9wg krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.65@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:48:01.487016 13022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:48:01.474721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:46457 (local address 127.3.243.65:43787)
0619 18:48:01.475011 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0619 18:48:01.475035 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 18:48:01.475161 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:48:01.475528 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:48:01.475533 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:48:01.476065 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:48:01.476337 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:01.476351 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:01.477848 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:01.477853 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:01.479543 (+ 1690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:01.479560 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:01.480713 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:01.480720 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:01.480927 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:01.481700 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:48:01.481725 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:48:01.486679 (+ 4954us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.243.126:46457: Server oryx/127.3.243.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157}
W20260619 18:48:01.487214 13023 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:48:01.474877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.65:43787 (local address 127.3.243.126:46457)
0619 18:48:01.475052 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 18:48:01.475059 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:48:01.475104 (+ 45us) server_negotiation.cc:407] Connection header received
0619 18:48:01.475290 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:48:01.475296 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:48:01.475387 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:48:01.475493 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:48:01.476480 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:01.477702 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:01.479732 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:01.480522 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:01.480639 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:01.486939 (+ 6300us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.243.65:43787: BlockingRecv error: recv got EOF from 127.3.243.65:43787 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 19 18:48:01 dist-test-slave-m9wg krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.65@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:48:01 dist-test-slave-m9wg krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.65@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM, Server not found in Kerberos database
W20260619 18:48:01.504693 13023 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:48:01.488143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.65:38133 (local address 127.3.243.126:46457)
0619 18:48:01.488321 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 18:48:01.488327 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:48:01.492769 (+ 4442us) server_negotiation.cc:407] Connection header received
0619 18:48:01.493001 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:48:01.493007 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:48:01.493089 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:48:01.493158 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:48:01.494277 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:01.495475 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:01.497640 (+ 2165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:01.498333 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:01.498450 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:01.504409 (+ 5959us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.243.65:38133: BlockingRecv error: recv got EOF from 127.3.243.65:38133 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 18:48:01.505259 13022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:48:01.487742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:46457 (local address 127.3.243.65:38133)
0619 18:48:01.492679 (+ 4937us) negotiation.cc:107] Waiting for socket to connect
0619 18:48:01.492700 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 18:48:01.492870 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:48:01.493373 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:48:01.493378 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:48:01.493869 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:48:01.494091 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:01.494104 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:01.495680 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:01.495686 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:01.497464 (+ 1778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:01.497480 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:01.498732 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:01.498738 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:01.498941 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:01.499716 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:48:01.499740 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:48:01.504913 (+ 5173us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.243.126:46457: Server oryx/127.3.243.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":4822,"thread_start_us":11771,"threads_started":1}
W20260619 18:48:01.505707 13020 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 18:48:01.877977 13155 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.136s user 0.004s sys 0.010s
W20260619 18:48:01.879379 13156 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:48:00.729957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.66:42933 (local address 127.3.243.126:46457)
0619 18:48:00.730831 (+ 874us) server_negotiation.cc:206] Beginning negotiation
0619 18:48:00.730837 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:48:00.730858 (+ 21us) server_negotiation.cc:407] Connection header received
0619 18:48:00.732175 (+ 1317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:48:00.732181 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:48:00.732264 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:48:00.732364 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:48:00.733858 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:00.734579 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:00.736182 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:00.736595 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:00.736706 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:01.879159 (+1142453us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.243.66:42933: BlockingRecv error: recv got EOF from 127.3.243.66:42933 (error 108)
Metrics: {"server-negotiator.queue_time_us":752,"thread_start_us":620,"threads_started":1}
I20260619 18:48:01.880379 13155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:48:00.729372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:46457 (local address 127.3.243.66:42933)
0619 18:48:00.730107 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0619 18:48:00.730156 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 18:48:00.731893 (+ 1737us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:48:00.732631 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:48:00.732639 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:48:00.733124 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:48:00.733703 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:00.733721 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:00.734720 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:00.734724 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:00.735994 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:00.736007 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:00.736814 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:00.736821 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:00.737059 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:00.741498 (+ 4439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:48:00.741525 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:48:01.879021 (+1137496us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.243.126:46457: Server oryx/127.3.243.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":289,"threads_started":1}
W20260619 18:48:01.881311 13153 heartbeater.cc:644] Failed to heartbeat to 127.3.243.126:46457 (0 consecutive failures): Not authorized: Failed to ping master at 127.3.243.126:46457: Client connection negotiation failed: client connection to 127.3.243.126:46457: Server oryx/127.3.243.126@KRBTEST.COM not found in Kerberos database
Jun 19 18:48:01 dist-test-slave-m9wg krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.66@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:48:01 dist-test-slave-m9wg krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.66@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM, Server not found in Kerberos database
W20260619 18:48:01.892199 13156 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:48:01.881981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.66:36505 (local address 127.3.243.126:46457)
0619 18:48:01.882121 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 18:48:01.882125 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:48:01.882189 (+ 64us) server_negotiation.cc:407] Connection header received
0619 18:48:01.882363 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:48:01.882367 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:48:01.882426 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:48:01.882502 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:48:01.883443 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:01.884174 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:01.886151 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:01.886544 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:01.886630 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:01.892005 (+ 5375us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.243.66:36505: BlockingRecv error: recv got EOF from 127.3.243.66:36505 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 18:48:01.892206 13155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:48:01.881871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:46457 (local address 127.3.243.66:36505)
0619 18:48:01.882088 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0619 18:48:01.882120 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0619 18:48:01.882264 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:48:01.882547 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:48:01.882552 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:48:01.883066 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:48:01.883311 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:01.883324 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:01.884302 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:01.884309 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:01.886002 (+ 1693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:01.886018 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:01.886701 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:01.886708 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:01.886891 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:01.887536 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:48:01.887562 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:48:01.891843 (+ 4281us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.243.126:46457: Server oryx/127.3.243.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":60}
Jun 19 18:48:01 dist-test-slave-m9wg krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.66@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:48:01 dist-test-slave-m9wg krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.66@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM, Server not found in Kerberos database
W20260619 18:48:01.902784 13156 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:48:01.893162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.66:53393 (local address 127.3.243.126:46457)
0619 18:48:01.893293 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 18:48:01.893296 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:48:01.893331 (+ 35us) server_negotiation.cc:407] Connection header received
0619 18:48:01.893482 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:48:01.893486 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:48:01.893549 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:48:01.893628 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:48:01.894499 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:01.895215 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:01.897163 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:01.897546 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:01.897636 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:01.902602 (+ 4966us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.243.66:53393: BlockingRecv error: recv got EOF from 127.3.243.66:53393 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 18:48:01.903185 13155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:48:01.893018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:46457 (local address 127.3.243.66:53393)
0619 18:48:01.893265 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0619 18:48:01.893284 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 18:48:01.893394 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:48:01.893675 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:48:01.893681 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:48:01.894112 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:48:01.894325 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:01.894337 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:01.895345 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:01.895351 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:01.897005 (+ 1654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:01.897022 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:01.897689 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:01.897696 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:01.897894 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:01.898490 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:48:01.898514 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:48:01.902977 (+ 4463us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.243.126:46457: Server oryx/127.3.243.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136}
W20260619 18:48:01.903457 13153 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 18:48:02.328787 13288 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.037s user 0.010s sys 0.010s
W20260619 18:48:02.329829 13290 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:48:01.231779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.67:33291 (local address 127.3.243.126:46457)
0619 18:48:01.265821 (+ 34042us) server_negotiation.cc:206] Beginning negotiation
0619 18:48:01.265829 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:48:01.272216 (+ 6387us) server_negotiation.cc:407] Connection header received
0619 18:48:01.273611 (+ 1395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:48:01.273619 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:48:01.273721 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:48:01.273812 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:48:01.277656 (+ 3844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:01.279092 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:01.284295 (+ 5203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:01.285018 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:01.285126 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:02.329629 (+1044503us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.243.67:33291: BlockingRecv error: recv got EOF from 127.3.243.67:33291 (error 108)
Metrics: {"server-negotiator.queue_time_us":33863,"thread_start_us":36520,"threads_started":1}
I20260619 18:48:02.330317 13288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:48:01.231076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:46457 (local address 127.3.243.67:33291)
0619 18:48:01.269078 (+ 38002us) negotiation.cc:107] Waiting for socket to connect
0619 18:48:01.269156 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0619 18:48:01.272056 (+ 2900us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:48:01.274620 (+ 2564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:48:01.274636 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:48:01.275538 (+ 902us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:48:01.276545 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:01.276573 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:01.281725 (+ 5152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:01.281731 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:01.284076 (+ 2345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:01.284095 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:01.285539 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:01.285547 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:01.285889 (+ 342us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:01.291920 (+ 6031us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:48:01.291968 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:48:02.329445 (+1037477us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.243.126:46457: Server oryx/127.3.243.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":37554,"thread_start_us":41030,"threads_started":1}
W20260619 18:48:02.331107 13286 heartbeater.cc:644] Failed to heartbeat to 127.3.243.126:46457 (0 consecutive failures): Not authorized: Failed to ping master at 127.3.243.126:46457: Client connection negotiation failed: client connection to 127.3.243.126:46457: Server oryx/127.3.243.126@KRBTEST.COM not found in Kerberos database
Jun 19 18:48:02 dist-test-slave-m9wg krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.67@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:48:02 dist-test-slave-m9wg krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.67@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:48:02.339731 13288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:48:02.331645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:46457 (local address 127.3.243.67:45779)
0619 18:48:02.331818 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0619 18:48:02.331834 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 18:48:02.331931 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:48:02.332241 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:48:02.332244 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:48:02.332579 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:48:02.332753 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:02.332761 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:02.333907 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:02.333911 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:02.334972 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:02.334985 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:02.335629 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:02.335634 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:02.335745 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:02.336320 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:48:02.336337 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:48:02.339540 (+ 3203us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.243.126:46457: Server oryx/127.3.243.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":59}
W20260619 18:48:02.339903 13290 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:48:02.331709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.67:45779 (local address 127.3.243.126:46457)
0619 18:48:02.331842 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 18:48:02.331846 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:48:02.331875 (+ 29us) server_negotiation.cc:407] Connection header received
0619 18:48:02.332040 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:48:02.332043 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:48:02.332102 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:48:02.332225 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:48:02.332856 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:02.333791 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:02.335090 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:02.335465 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:02.335591 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:02.339717 (+ 4126us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.243.67:45779: BlockingRecv error: recv got EOF from 127.3.243.67:45779 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Jun 19 18:48:02 dist-test-slave-m9wg krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.67@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:48:02 dist-test-slave-m9wg krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.67@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:48:02.347479 13288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:48:02.340320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:46457 (local address 127.3.243.67:50765)
0619 18:48:02.340450 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0619 18:48:02.340463 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 18:48:02.340542 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:48:02.340836 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:48:02.340838 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:48:02.341104 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:48:02.341238 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:02.341246 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:02.342119 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:02.342122 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:02.342994 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:02.343013 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:02.343586 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:02.343590 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:02.343704 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:02.344179 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:48:02.344201 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:48:02.347301 (+ 3100us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.243.126:46457: Server oryx/127.3.243.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":44}
W20260619 18:48:02.347681 13290 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:48:02.340389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.67:50765 (local address 127.3.243.126:46457)
0619 18:48:02.340617 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0619 18:48:02.340621 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:48:02.340634 (+ 13us) server_negotiation.cc:407] Connection header received
0619 18:48:02.340670 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:48:02.340673 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:48:02.340732 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:48:02.340800 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:48:02.341333 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:02.342020 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:02.343096 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:02.343488 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:02.343570 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:02.347469 (+ 3899us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.243.67:50765: BlockingRecv error: recv got EOF from 127.3.243.67:50765 (error 108)
Metrics: {"server-negotiator.queue_time_us":133}
W20260619 18:48:02.347728 13286 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 19 18:48:02 dist-test-slave-m9wg krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.65@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:48:02 dist-test-slave-m9wg krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.65@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:48:02.514843 13294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:48:02.506262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:46457 (local address 127.3.243.65:47879)
0619 18:48:02.506751 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0619 18:48:02.506767 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 18:48:02.506945 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:48:02.507524 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:48:02.507527 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:48:02.508058 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:48:02.508289 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:02.508300 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:02.509274 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:02.509278 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:02.510222 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:02.510233 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:02.510863 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:02.510867 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:02.511001 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:02.511510 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:48:02.511526 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:48:02.514620 (+ 3094us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.243.126:46457: Server oryx/127.3.243.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":281,"threads_started":1}
W20260619 18:48:02.514968 13290 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:48:02.506357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.65:47879 (local address 127.3.243.126:46457)
0619 18:48:02.506510 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 18:48:02.506514 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:48:02.506846 (+ 332us) server_negotiation.cc:407] Connection header received
0619 18:48:02.507111 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:48:02.507117 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:48:02.507206 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:48:02.507277 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:48:02.508440 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:02.509147 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:02.510335 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:02.510737 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:02.510825 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:02.514798 (+ 3973us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.243.65:47879: BlockingRecv error: recv got EOF from 127.3.243.65:47879 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 19 18:48:02 dist-test-slave-m9wg krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.66@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:48:02 dist-test-slave-m9wg krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.66@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:48:02.912374 13295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:48:02.904012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:46457 (local address 127.3.243.66:51759)
0619 18:48:02.904558 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0619 18:48:02.904577 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 18:48:02.904717 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:48:02.905079 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:48:02.905084 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:48:02.905605 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:48:02.905758 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:02.905766 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:02.906768 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:02.906772 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:02.907729 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:02.907742 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:02.908508 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:02.908513 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:02.908631 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:02.909162 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:48:02.909179 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:48:02.912130 (+ 2951us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.243.126:46457: Server oryx/127.3.243.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":334,"threads_started":1}
W20260619 18:48:02.912565 13290 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:48:02.904107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.66:51759 (local address 127.3.243.126:46457)
0619 18:48:02.904281 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 18:48:02.904287 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:48:02.904662 (+ 375us) server_negotiation.cc:407] Connection header received
0619 18:48:02.904833 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:48:02.904838 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:48:02.904929 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:48:02.905028 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:48:02.905876 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:02.906661 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:02.907845 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:02.908358 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:02.908458 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:02.912354 (+ 3896us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.243.66:51759: BlockingRecv error: recv got EOF from 127.3.243.66:51759 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 19 18:48:03 dist-test-slave-m9wg krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.67@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:48:03 dist-test-slave-m9wg krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.67@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:48:03.356833 13296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:48:03.348291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:46457 (local address 127.3.243.67:51247)
0619 18:48:03.348810 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0619 18:48:03.348826 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 18:48:03.348984 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:48:03.349315 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:48:03.349319 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:48:03.349715 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:48:03.349864 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:03.349873 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:03.350833 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:03.350837 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:03.351773 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:03.351784 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:03.352581 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:03.352587 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:03.352736 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:03.353288 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:48:03.353305 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:48:03.356521 (+ 3216us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.243.126:46457: Server oryx/127.3.243.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":291,"threads_started":1}
W20260619 18:48:03.356987 13290 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:48:03.348383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.67:51247 (local address 127.3.243.126:46457)
0619 18:48:03.348572 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 18:48:03.348578 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:48:03.348905 (+ 327us) server_negotiation.cc:407] Connection header received
0619 18:48:03.349106 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:48:03.349112 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:48:03.349184 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:48:03.349263 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:48:03.350009 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:03.350708 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:03.351909 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:03.352412 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:03.352519 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:03.356742 (+ 4223us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.243.67:51247: BlockingRecv error: recv got EOF from 127.3.243.67:51247 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 19 18:48:03 dist-test-slave-m9wg krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.65@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:48:03 dist-test-slave-m9wg krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.65@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:48:03.524394 13297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:48:03.515794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:46457 (local address 127.3.243.65:53187)
0619 18:48:03.516399 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0619 18:48:03.516417 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 18:48:03.516533 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:48:03.516789 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:48:03.516792 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:48:03.517184 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:48:03.517362 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:03.517372 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:03.518343 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:03.518347 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:03.519271 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:03.519285 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:03.520058 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:03.520066 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:03.520277 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:03.520922 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:48:03.520946 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:48:03.524108 (+ 3162us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.243.126:46457: Server oryx/127.3.243.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":333,"threads_started":1}
W20260619 18:48:03.524582 13290 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:48:03.515861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.65:53187 (local address 127.3.243.126:46457)
0619 18:48:03.516036 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 18:48:03.516040 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:48:03.516481 (+ 441us) server_negotiation.cc:407] Connection header received
0619 18:48:03.516617 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:48:03.516621 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:48:03.516682 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:48:03.516760 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:48:03.517483 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:03.518180 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:03.519433 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:03.519872 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:03.520013 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:03.524359 (+ 4346us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.243.65:53187: BlockingRecv error: recv got EOF from 127.3.243.65:53187 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 19 18:48:03 dist-test-slave-m9wg krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.66@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:48:03 dist-test-slave-m9wg krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.66@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:48:03.921936 13298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:48:03.913257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:46457 (local address 127.3.243.66:47619)
0619 18:48:03.913708 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0619 18:48:03.913722 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 18:48:03.913851 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:48:03.914156 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:48:03.914161 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:48:03.914624 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:48:03.914794 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:03.914803 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:03.915878 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:03.915884 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:03.917028 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:03.917046 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:03.917683 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:03.917688 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:03.917804 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:03.918408 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:48:03.918426 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:48:03.921609 (+ 3183us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.243.126:46457: Server oryx/127.3.243.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":250,"threads_started":1}
W20260619 18:48:03.922022 13290 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:48:03.913298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.66:47619 (local address 127.3.243.126:46457)
0619 18:48:03.913470 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 18:48:03.913476 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:48:03.913806 (+ 330us) server_negotiation.cc:407] Connection header received
0619 18:48:03.913961 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:48:03.913966 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:48:03.914029 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:48:03.914110 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:48:03.914935 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:03.915690 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:03.917154 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:03.917568 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:03.917659 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:03.921819 (+ 4160us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.243.66:47619: BlockingRecv error: recv got EOF from 127.3.243.66:47619 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 19 18:48:04 dist-test-slave-m9wg krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.67@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:48:04 dist-test-slave-m9wg krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.67@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:48:04.365919 13299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:48:04.357762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:46457 (local address 127.3.243.67:36033)
0619 18:48:04.358249 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0619 18:48:04.358264 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 18:48:04.358373 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:48:04.358616 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:48:04.358618 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:48:04.359012 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:48:04.359191 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:04.359200 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:04.360213 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:04.360217 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:04.361148 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:04.361162 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:04.361774 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:04.361778 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:04.361895 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:04.362397 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:48:04.362413 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:48:04.365660 (+ 3247us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.243.126:46457: Server oryx/127.3.243.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":257,"threads_started":1}
W20260619 18:48:04.366079 13290 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:48:04.357876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.67:36033 (local address 127.3.243.126:46457)
0619 18:48:04.358040 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 18:48:04.358045 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:48:04.358325 (+ 280us) server_negotiation.cc:407] Connection header received
0619 18:48:04.358463 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:48:04.358466 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:48:04.358525 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:48:04.358599 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:48:04.359322 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:04.360069 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:04.361267 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:04.361660 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:04.361751 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:04.365843 (+ 4092us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.243.67:36033: BlockingRecv error: recv got EOF from 127.3.243.67:36033 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 19 18:48:04 dist-test-slave-m9wg krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.65@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:48:04 dist-test-slave-m9wg krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.65@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:48:04.534199 13300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:48:04.525274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:46457 (local address 127.3.243.65:41083)
0619 18:48:04.525805 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0619 18:48:04.525822 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 18:48:04.525946 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:48:04.526296 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:48:04.526300 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:48:04.526699 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:48:04.526910 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:04.526920 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:04.528005 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:04.528009 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:04.529018 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:04.529033 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:04.529682 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:04.529686 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:04.529822 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:04.530402 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:48:04.530420 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:48:04.533868 (+ 3448us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.243.126:46457: Server oryx/127.3.243.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":283,"threads_started":1}
W20260619 18:48:04.534350 13290 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:48:04.525340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.65:41083 (local address 127.3.243.126:46457)
0619 18:48:04.525494 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 18:48:04.525498 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:48:04.525886 (+ 388us) server_negotiation.cc:407] Connection header received
0619 18:48:04.526050 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:48:04.526053 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:48:04.526134 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:48:04.526213 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:48:04.527048 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:04.527861 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:04.529145 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:04.529566 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:04.529658 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:04.534126 (+ 4468us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.243.65:41083: BlockingRecv error: recv got EOF from 127.3.243.65:41083 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 19 18:48:04 dist-test-slave-m9wg krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.66@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:48:04 dist-test-slave-m9wg krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.66@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:48:04.931339 13301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:48:04.922859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:46457 (local address 127.3.243.66:52039)
0619 18:48:04.923332 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0619 18:48:04.923346 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 18:48:04.923474 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:48:04.923784 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:48:04.923789 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:48:04.924204 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:48:04.924363 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:04.924373 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:04.925367 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:04.925371 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:04.926392 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:04.926405 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:04.927228 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:04.927233 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:04.927492 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:04.927999 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:48:04.928022 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:48:04.931107 (+ 3085us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.243.126:46457: Server oryx/127.3.243.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":253,"threads_started":1}
W20260619 18:48:04.931491 13290 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:48:04.922937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.66:52039 (local address 127.3.243.126:46457)
0619 18:48:04.923124 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 18:48:04.923130 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:48:04.923438 (+ 308us) server_negotiation.cc:407] Connection header received
0619 18:48:04.923582 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:48:04.923586 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:48:04.923647 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:48:04.923730 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:48:04.924496 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:04.925224 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:04.926531 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:04.927075 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:04.927174 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:04.931291 (+ 4117us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.243.66:52039: BlockingRecv error: recv got EOF from 127.3.243.66:52039 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 19 18:48:05 dist-test-slave-m9wg krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.67@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:48:05 dist-test-slave-m9wg krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.67@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:48:05.375094 13302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:48:05.366767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:46457 (local address 127.3.243.67:45221)
0619 18:48:05.367208 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0619 18:48:05.367224 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 18:48:05.367336 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:48:05.367618 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:48:05.367621 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:48:05.367977 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:48:05.368179 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:05.368189 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:05.369168 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:05.369171 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:05.370063 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:05.370074 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:05.370682 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:05.370686 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:05.370806 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:05.371375 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:48:05.371392 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:48:05.374844 (+ 3452us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.243.126:46457: Server oryx/127.3.243.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":225,"threads_started":1}
W20260619 18:48:05.375190 13290 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:48:05.366861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.67:45221 (local address 127.3.243.126:46457)
0619 18:48:05.367061 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 18:48:05.367067 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:48:05.367295 (+ 228us) server_negotiation.cc:407] Connection header received
0619 18:48:05.367430 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:48:05.367434 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:48:05.367510 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:48:05.367590 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:48:05.368307 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:05.369062 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:05.370183 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:05.370579 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:05.370667 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:05.375016 (+ 4349us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.243.67:45221: BlockingRecv error: recv got EOF from 127.3.243.67:45221 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Jun 19 18:48:05 dist-test-slave-m9wg krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.65@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:48:05 dist-test-slave-m9wg krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.65@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:48:05.543603 13303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:48:05.535093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:46457 (local address 127.3.243.65:53593)
0619 18:48:05.535598 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0619 18:48:05.535614 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 18:48:05.535753 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:48:05.536068 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:48:05.536072 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:48:05.536523 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:48:05.536683 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:05.536692 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:05.537706 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:05.537711 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:05.538731 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:05.538744 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:05.539458 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:05.539463 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:05.539639 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:05.540267 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:48:05.540285 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:48:05.543365 (+ 3080us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.243.126:46457: Server oryx/127.3.243.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":262,"threads_started":1}
W20260619 18:48:05.543743 13290 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:48:05.535163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.65:53593 (local address 127.3.243.126:46457)
0619 18:48:05.535355 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 18:48:05.535359 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:48:05.535709 (+ 350us) server_negotiation.cc:407] Connection header received
0619 18:48:05.535871 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:48:05.535875 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:48:05.535935 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:48:05.536062 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:48:05.536822 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:05.537574 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:05.538865 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:05.539321 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:05.539427 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:05.543539 (+ 4112us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.243.65:53593: BlockingRecv error: recv got EOF from 127.3.243.65:53593 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Jun 19 18:48:05 dist-test-slave-m9wg krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.66@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:48:05 dist-test-slave-m9wg krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.66@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:48:05.945441 13304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:48:05.932276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:46457 (local address 127.3.243.66:36429)
0619 18:48:05.932805 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0619 18:48:05.932820 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 18:48:05.932929 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:48:05.933232 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:48:05.933237 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:48:05.933632 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:48:05.933788 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:05.933797 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:05.934751 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:05.934756 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:05.935722 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:05.935733 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:05.936568 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:05.936574 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:05.936739 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:05.941847 (+ 5108us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:48:05.941866 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:48:05.945198 (+ 3332us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.243.126:46457: Server oryx/127.3.243.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":310,"threads_started":1}
W20260619 18:48:05.945583 13290 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:48:05.932333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.66:36429 (local address 127.3.243.126:46457)
0619 18:48:05.932484 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 18:48:05.932489 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:48:05.932897 (+ 408us) server_negotiation.cc:407] Connection header received
0619 18:48:05.933010 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:48:05.933013 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:48:05.933075 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:48:05.933176 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:48:05.933925 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:05.934629 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:05.935859 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:05.936434 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:05.936540 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:05.945387 (+ 8847us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.243.66:36429: BlockingRecv error: recv got EOF from 127.3.243.66:36429 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Jun 19 18:48:06 dist-test-slave-m9wg krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.67@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:48:06 dist-test-slave-m9wg krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.67@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:48:06.384459 13305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:48:06.375990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:46457 (local address 127.3.243.67:50223)
0619 18:48:06.376512 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0619 18:48:06.376527 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 18:48:06.376674 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:48:06.377034 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:48:06.377038 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:48:06.377416 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:48:06.377571 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:06.377581 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:06.378555 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:06.378560 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:06.379568 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:06.379581 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:06.380260 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:06.380266 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:06.380424 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:06.380984 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:48:06.381001 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:48:06.384183 (+ 3182us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.243.126:46457: Server oryx/127.3.243.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":281,"threads_started":1}
W20260619 18:48:06.384626 13290 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:48:06.376103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.67:50223 (local address 127.3.243.126:46457)
0619 18:48:06.376309 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0619 18:48:06.376315 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:48:06.376621 (+ 306us) server_negotiation.cc:407] Connection header received
0619 18:48:06.376796 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:48:06.376802 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:48:06.376890 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:48:06.376972 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:48:06.377712 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:06.378425 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:06.379704 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:06.380091 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:06.380226 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:06.384381 (+ 4155us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.243.67:50223: BlockingRecv error: recv got EOF from 127.3.243.67:50223 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Jun 19 18:48:06 dist-test-slave-m9wg krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.65@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:48:06 dist-test-slave-m9wg krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.65@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:48:06.553237 13306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:48:06.544450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:46457 (local address 127.3.243.65:49411)
0619 18:48:06.544966 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0619 18:48:06.544981 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 18:48:06.545112 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:48:06.545420 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:48:06.545425 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:48:06.545857 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:48:06.546084 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:06.546094 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:06.547066 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:06.547071 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:06.548011 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:06.548024 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:06.548726 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:06.548733 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:06.548910 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:06.549416 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:48:06.549433 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:48:06.552970 (+ 3537us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.243.126:46457: Server oryx/127.3.243.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":270,"threads_started":1}
W20260619 18:48:06.553438 13290 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:48:06.544590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.65:49411 (local address 127.3.243.126:46457)
0619 18:48:06.544776 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 18:48:06.544783 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:48:06.545063 (+ 280us) server_negotiation.cc:407] Connection header received
0619 18:48:06.545223 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:48:06.545227 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:48:06.545289 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:48:06.545367 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:48:06.546230 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:06.546935 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:06.548172 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:06.548580 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:06.548689 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:06.553169 (+ 4480us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.243.65:49411: BlockingRecv error: recv got EOF from 127.3.243.65:49411 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 19 18:48:06 dist-test-slave-m9wg krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.66@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:48:06 dist-test-slave-m9wg krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.66@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:48:06.954607 13307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:48:06.946255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:46457 (local address 127.3.243.66:40697)
0619 18:48:06.946732 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0619 18:48:06.946749 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 18:48:06.946871 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:48:06.947183 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:48:06.947187 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:48:06.947631 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:48:06.947773 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:06.947782 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:06.948818 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:06.948824 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:06.949735 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:06.949746 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:06.950424 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:06.950430 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:06.950618 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:06.951157 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:48:06.951174 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:48:06.954278 (+ 3104us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.243.126:46457: Server oryx/127.3.243.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":236,"threads_started":1}
W20260619 18:48:06.954712 13290 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:48:06.946468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.66:40697 (local address 127.3.243.126:46457)
0619 18:48:06.946671 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 18:48:06.946677 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:48:06.946820 (+ 143us) server_negotiation.cc:407] Connection header received
0619 18:48:06.946981 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:48:06.946998 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:48:06.947056 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:48:06.947135 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:48:06.947910 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:06.948696 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:06.949866 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:06.950289 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:06.950398 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:06.954479 (+ 4081us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.243.66:40697: BlockingRecv error: recv got EOF from 127.3.243.66:40697 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Jun 19 18:48:07 dist-test-slave-m9wg krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.67@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:48:07 dist-test-slave-m9wg krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.67@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:48:07.393793 13308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:48:07.385375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:46457 (local address 127.3.243.67:52339)
0619 18:48:07.385848 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0619 18:48:07.385866 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 18:48:07.386031 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:48:07.386406 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:48:07.386410 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:48:07.386917 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:48:07.387112 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:07.387126 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:07.388225 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:07.388229 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:07.389142 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:07.389153 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:07.389839 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:07.389842 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:07.389970 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:07.390515 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:48:07.390533 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:48:07.393568 (+ 3035us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.243.126:46457: Server oryx/127.3.243.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":308,"threads_started":1}
W20260619 18:48:07.394021 13290 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:48:07.385477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.67:52339 (local address 127.3.243.126:46457)
0619 18:48:07.385677 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 18:48:07.385682 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:48:07.385962 (+ 280us) server_negotiation.cc:407] Connection header received
0619 18:48:07.386155 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:48:07.386161 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:48:07.386263 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:48:07.386382 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:48:07.387272 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:07.388070 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:07.389288 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:07.389720 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:07.389817 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:07.393756 (+ 3939us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.243.67:52339: BlockingRecv error: recv got EOF from 127.3.243.67:52339 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Jun 19 18:48:07 dist-test-slave-m9wg krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.65@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:48:07 dist-test-slave-m9wg krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.65@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:48:07.562645 13309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:48:07.554294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:46457 (local address 127.3.243.65:33213)
0619 18:48:07.554718 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0619 18:48:07.554734 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 18:48:07.554849 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:48:07.555121 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:48:07.555124 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:48:07.555522 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:48:07.555683 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:07.555692 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:07.556654 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:07.556658 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:07.557631 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:07.557643 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:07.558296 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:07.558300 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:07.558426 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:07.558971 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:48:07.558987 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:48:07.562335 (+ 3348us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.243.126:46457: Server oryx/127.3.243.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":215,"threads_started":1}
W20260619 18:48:07.562963 13290 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:48:07.554340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.65:33213 (local address 127.3.243.126:46457)
0619 18:48:07.554529 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 18:48:07.554536 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:48:07.554805 (+ 269us) server_negotiation.cc:407] Connection header received
0619 18:48:07.554937 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:48:07.554941 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:48:07.555015 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:48:07.555089 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:48:07.555789 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:07.556549 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:07.557757 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:07.558182 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:07.558269 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:07.562585 (+ 4316us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.243.65:33213: BlockingRecv error: recv got EOF from 127.3.243.65:33213 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 19 18:48:07 dist-test-slave-m9wg krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.66@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:48:07 dist-test-slave-m9wg krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.66@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:48:07.963815 13310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:48:07.955517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:46457 (local address 127.3.243.66:33107)
0619 18:48:07.955917 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0619 18:48:07.955933 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 18:48:07.956047 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:48:07.956320 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:48:07.956322 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:48:07.956709 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:48:07.956883 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:07.956892 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:07.957833 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:07.957836 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:07.958729 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:07.958740 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:07.959385 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:07.959388 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:07.959505 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:07.960066 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:48:07.960083 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:48:07.963597 (+ 3514us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.243.126:46457: Server oryx/127.3.243.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":196,"threads_started":1}
W20260619 18:48:07.964031 13290 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:48:07.955580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.66:33107 (local address 127.3.243.126:46457)
0619 18:48:07.955802 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0619 18:48:07.955822 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 18:48:07.956002 (+ 180us) server_negotiation.cc:407] Connection header received
0619 18:48:07.956142 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:48:07.956164 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:48:07.956225 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:48:07.956293 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:48:07.957022 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:07.957725 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:07.958832 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:07.959271 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:07.959361 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:07.963783 (+ 4422us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.243.66:33107: BlockingRecv error: recv got EOF from 127.3.243.66:33107 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Jun 19 18:48:08 dist-test-slave-m9wg krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.67@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:48:08 dist-test-slave-m9wg krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.67@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:48:08.402968 13311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:48:08.394661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:46457 (local address 127.3.243.67:32809)
0619 18:48:08.395186 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0619 18:48:08.395203 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 18:48:08.395341 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:48:08.395679 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:48:08.395684 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:48:08.396107 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:48:08.396322 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:08.396332 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:08.397340 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:08.397346 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:08.398268 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:08.398279 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:08.398954 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:08.398960 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:08.399126 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:08.399675 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:48:08.399692 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:48:08.402745 (+ 3053us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.243.126:46457: Server oryx/127.3.243.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":274,"threads_started":1}
W20260619 18:48:08.403190 13290 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:48:08.394758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.67:32809 (local address 127.3.243.126:46457)
0619 18:48:08.395000 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0619 18:48:08.395007 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:48:08.395286 (+ 279us) server_negotiation.cc:407] Connection header received
0619 18:48:08.395460 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:48:08.395466 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:48:08.395546 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:48:08.395624 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:48:08.396466 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:08.397208 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:08.398408 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:08.398808 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:08.398913 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:08.402960 (+ 4047us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.243.67:32809: BlockingRecv error: recv got EOF from 127.3.243.67:32809 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Jun 19 18:48:08 dist-test-slave-m9wg krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.65@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:48:08 dist-test-slave-m9wg krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.65@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:48:08.572074 13312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:48:08.563601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:46457 (local address 127.3.243.65:34427)
0619 18:48:08.564144 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0619 18:48:08.564180 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 18:48:08.564295 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:48:08.564563 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:48:08.564566 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:48:08.564965 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:48:08.565149 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:08.565159 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:08.566054 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:08.566057 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:08.566965 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:08.566990 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:08.567678 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:08.567681 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:08.567789 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:08.568371 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:48:08.568390 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:48:08.571748 (+ 3358us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.243.126:46457: Server oryx/127.3.243.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":256,"threads_started":1}
W20260619 18:48:08.572193 13290 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:48:08.563695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.65:34427 (local address 127.3.243.126:46457)
0619 18:48:08.563948 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0619 18:48:08.563953 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:48:08.564257 (+ 304us) server_negotiation.cc:407] Connection header received
0619 18:48:08.564390 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:48:08.564393 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:48:08.564457 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:48:08.564534 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:48:08.565265 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:08.565947 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:08.567109 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:08.567549 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:08.567656 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:08.571975 (+ 4319us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.243.65:34427: BlockingRecv error: recv got EOF from 127.3.243.65:34427 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Jun 19 18:48:08 dist-test-slave-m9wg krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.66@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:48:08 dist-test-slave-m9wg krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.66@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:48:08.973541 13313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:48:08.964734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:46457 (local address 127.3.243.66:55975)
0619 18:48:08.965265 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0619 18:48:08.965282 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 18:48:08.965395 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:48:08.965654 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:48:08.965657 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:48:08.966096 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:48:08.966268 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:08.966278 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:08.967286 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:08.967290 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:08.968274 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:08.968286 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:08.968975 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:08.968979 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:08.969091 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:08.969604 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:48:08.969621 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:48:08.973281 (+ 3660us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.243.126:46457: Server oryx/127.3.243.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":258,"threads_started":1}
W20260619 18:48:08.973671 13290 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:48:08.964808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.66:55975 (local address 127.3.243.126:46457)
0619 18:48:08.965042 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0619 18:48:08.965048 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:48:08.965356 (+ 308us) server_negotiation.cc:407] Connection header received
0619 18:48:08.965483 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:48:08.965487 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:48:08.965548 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:48:08.965625 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:48:08.966412 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:08.967174 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:08.968408 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:08.968839 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:08.968940 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:08.973460 (+ 4520us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.243.66:55975: BlockingRecv error: recv got EOF from 127.3.243.66:55975 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Jun 19 18:48:09 dist-test-slave-m9wg krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.67@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:48:09 dist-test-slave-m9wg krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.67@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:48:09.412482 13314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:48:09.403929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:46457 (local address 127.3.243.67:48017)
0619 18:48:09.404531 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0619 18:48:09.404547 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 18:48:09.404660 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:48:09.404999 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:48:09.405004 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:48:09.405508 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:48:09.405683 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:09.405695 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:09.406822 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:09.406826 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:09.407791 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:09.407804 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:09.408513 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:09.408517 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:09.408644 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:09.409161 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:48:09.409178 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:48:09.412222 (+ 3044us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.243.126:46457: Server oryx/127.3.243.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":290,"threads_started":1}
W20260619 18:48:09.412560 13290 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:48:09.403987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.67:48017 (local address 127.3.243.126:46457)
0619 18:48:09.404178 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 18:48:09.404184 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:48:09.404619 (+ 435us) server_negotiation.cc:407] Connection header received
0619 18:48:09.404774 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:48:09.404780 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:48:09.404864 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:48:09.404945 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:48:09.405832 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:09.406700 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:09.407898 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:09.408384 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:09.408504 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:09.412383 (+ 3879us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.243.67:48017: BlockingRecv error: recv got EOF from 127.3.243.67:48017 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 19 18:48:09 dist-test-slave-m9wg krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.65@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:48:09 dist-test-slave-m9wg krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.65@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:48:09.581722 13315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:48:09.573153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:46457 (local address 127.3.243.65:37757)
0619 18:48:09.573680 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0619 18:48:09.573696 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 18:48:09.573861 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:48:09.574175 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:48:09.574180 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:48:09.574595 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:48:09.574758 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:09.574768 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:09.575770 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:09.575776 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:09.576747 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:09.576759 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:09.577509 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:09.577515 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:09.577668 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:09.578226 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:48:09.578243 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:48:09.581397 (+ 3154us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.243.126:46457: Server oryx/127.3.243.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":245,"threads_started":1}
W20260619 18:48:09.581892 13290 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:48:09.573350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.65:37757 (local address 127.3.243.126:46457)
0619 18:48:09.573579 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0619 18:48:09.573585 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:48:09.573776 (+ 191us) server_negotiation.cc:407] Connection header received
0619 18:48:09.573984 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:48:09.573987 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:48:09.574049 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:48:09.574127 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:48:09.574909 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:09.575633 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:09.576893 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:09.577332 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:09.577446 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:09.581661 (+ 4215us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.243.65:37757: BlockingRecv error: recv got EOF from 127.3.243.65:37757 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Jun 19 18:48:09 dist-test-slave-m9wg krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.66@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:48:09 dist-test-slave-m9wg krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.66@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:48:09.982566 13316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:48:09.974376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:46457 (local address 127.3.243.66:60737)
0619 18:48:09.974877 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0619 18:48:09.974892 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 18:48:09.975005 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:48:09.975276 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:48:09.975279 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:48:09.975674 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:48:09.975856 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:09.975866 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:09.976961 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:09.976965 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:09.977875 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:09.977886 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:09.978512 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:09.978516 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:09.978649 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:09.979260 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:48:09.979277 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:48:09.982319 (+ 3042us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.243.126:46457: Server oryx/127.3.243.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":266,"threads_started":1}
W20260619 18:48:09.982728 13290 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:48:09.974506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.66:60737 (local address 127.3.243.126:46457)
0619 18:48:09.974661 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 18:48:09.974665 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:48:09.974955 (+ 290us) server_negotiation.cc:407] Connection header received
0619 18:48:09.975091 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:48:09.975094 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:48:09.975171 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:48:09.975248 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:48:09.975965 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:09.976833 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:09.977978 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:09.978376 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:09.978462 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:09.982530 (+ 4068us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.243.66:60737: BlockingRecv error: recv got EOF from 127.3.243.66:60737 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 19 18:48:10 dist-test-slave-m9wg krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.67@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:48:10 dist-test-slave-m9wg krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.67@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:48:10.421548 13317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:48:10.413334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:46457 (local address 127.3.243.67:51895)
0619 18:48:10.413840 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0619 18:48:10.413856 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 18:48:10.413976 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:48:10.414229 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:48:10.414232 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:48:10.414606 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:48:10.414763 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:10.414772 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:10.415709 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:10.415712 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:10.416672 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:10.416684 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:10.417355 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:10.417359 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:10.417473 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:10.418011 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:48:10.418029 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:48:10.421229 (+ 3200us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.243.126:46457: Server oryx/127.3.243.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":260,"threads_started":1}
W20260619 18:48:10.421664 13290 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:48:10.413372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.67:51895 (local address 127.3.243.126:46457)
0619 18:48:10.413520 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 18:48:10.413524 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:48:10.413917 (+ 393us) server_negotiation.cc:407] Connection header received
0619 18:48:10.414064 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:48:10.414068 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:48:10.414127 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:48:10.414201 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:48:10.414877 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:10.415604 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:10.416783 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:10.417225 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:10.417315 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:10.421434 (+ 4119us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.243.67:51895: BlockingRecv error: recv got EOF from 127.3.243.67:51895 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Jun 19 18:48:10 dist-test-slave-m9wg krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.65@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:48:10 dist-test-slave-m9wg krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.65@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:48:10.591388 13318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:48:10.582787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:46457 (local address 127.3.243.65:48707)
0619 18:48:10.583297 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0619 18:48:10.583317 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 18:48:10.583454 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:48:10.583795 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:48:10.583799 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:48:10.584315 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:48:10.584517 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:10.584531 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:10.585515 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:10.585518 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:10.586514 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:10.586529 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:10.587194 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:10.587198 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:10.587329 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:10.587895 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:48:10.587913 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:48:10.591073 (+ 3160us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.243.126:46457: Server oryx/127.3.243.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":278,"threads_started":1}
W20260619 18:48:10.591512 13290 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:48:10.582859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.65:48707 (local address 127.3.243.126:46457)
0619 18:48:10.583053 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 18:48:10.583059 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:48:10.583400 (+ 341us) server_negotiation.cc:407] Connection header received
0619 18:48:10.583576 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:48:10.583581 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:48:10.583652 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:48:10.583741 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:48:10.584682 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:10.585405 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:10.586658 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:10.587075 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:10.587169 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:10.591296 (+ 4127us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.243.65:48707: BlockingRecv error: recv got EOF from 127.3.243.65:48707 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 19 18:48:10 dist-test-slave-m9wg krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.66@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:48:10 dist-test-slave-m9wg krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.66@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:48:10.996381 13319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:48:10.983416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:46457 (local address 127.3.243.66:52947)
0619 18:48:10.983876 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0619 18:48:10.983891 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 18:48:10.984009 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:48:10.984327 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:48:10.984331 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:48:10.984761 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:48:10.984900 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:10.984909 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:10.985892 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:10.985896 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:10.986991 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:10.987005 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:10.987787 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:10.987791 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:10.987909 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:10.993015 (+ 5106us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:48:10.993034 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:48:10.996133 (+ 3099us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.243.126:46457: Server oryx/127.3.243.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":235,"threads_started":1}
W20260619 18:48:10.996569 13290 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:48:10.983487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.66:52947 (local address 127.3.243.126:46457)
0619 18:48:10.983663 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 18:48:10.983669 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:48:10.983987 (+ 318us) server_negotiation.cc:407] Connection header received
0619 18:48:10.984121 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:48:10.984125 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:48:10.984203 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:48:10.984281 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:48:10.985044 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:10.985764 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:10.987140 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:10.987664 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:10.987758 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:10.996339 (+ 8581us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.243.66:52947: BlockingRecv error: recv got EOF from 127.3.243.66:52947 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 19 18:48:11 dist-test-slave-m9wg krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.67@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:48:11 dist-test-slave-m9wg krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.67@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:48:11.442346 13320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:48:11.422403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:46457 (local address 127.3.243.67:37451)
0619 18:48:11.432599 (+ 10196us) negotiation.cc:107] Waiting for socket to connect
0619 18:48:11.432618 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 18:48:11.432729 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:48:11.433090 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:48:11.433095 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:48:11.433604 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:48:11.433796 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:11.433808 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:11.435230 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:11.435236 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:11.436805 (+ 1569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:11.436823 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:11.437567 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:11.437574 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:11.437758 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:11.438542 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:48:11.438565 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:48:11.442127 (+ 3562us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.243.126:46457: Server oryx/127.3.243.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":10082,"thread_start_us":9912,"threads_started":1}
W20260619 18:48:11.442490 13290 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:48:11.422733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.67:37451 (local address 127.3.243.126:46457)
0619 18:48:11.422952 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0619 18:48:11.422958 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:48:11.432686 (+ 9728us) server_negotiation.cc:407] Connection header received
0619 18:48:11.432831 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:48:11.432836 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:48:11.432933 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:48:11.433038 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:48:11.433935 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:11.435062 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:11.436935 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:11.437437 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:11.437547 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:11.442304 (+ 4757us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.243.67:37451: BlockingRecv error: recv got EOF from 127.3.243.67:37451 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 19 18:48:11 dist-test-slave-m9wg krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.65@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:48:11 dist-test-slave-m9wg krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.65@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:48:11.600809 13321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:48:11.592332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:46457 (local address 127.3.243.65:48263)
0619 18:48:11.592767 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0619 18:48:11.592813 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 18:48:11.592932 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:48:11.593191 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:48:11.593194 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:48:11.593557 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:48:11.593723 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:11.593732 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:11.594659 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:11.594663 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:11.595661 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:11.595673 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:11.596370 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:11.596375 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:11.596498 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:11.597036 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:48:11.597053 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:48:11.600497 (+ 3444us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.243.126:46457: Server oryx/127.3.243.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":225,"threads_started":1}
W20260619 18:48:11.600951 13290 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:48:11.592443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.65:48263 (local address 127.3.243.126:46457)
0619 18:48:11.592671 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0619 18:48:11.592677 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:48:11.592914 (+ 237us) server_negotiation.cc:407] Connection header received
0619 18:48:11.593031 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:48:11.593035 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:48:11.593098 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:48:11.593175 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:48:11.593827 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:11.594539 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:11.595765 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:11.596236 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:11.596318 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:11.600712 (+ 4394us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.243.65:48263: BlockingRecv error: recv got EOF from 127.3.243.65:48263 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
Jun 19 18:48:12 dist-test-slave-m9wg krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.66@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:48:12 dist-test-slave-m9wg krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.66@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:48:12.005559 13322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:48:11.997224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:46457 (local address 127.3.243.66:45441)
0619 18:48:11.997712 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0619 18:48:11.997728 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 18:48:11.997861 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:48:11.998176 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:48:11.998181 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:48:11.998584 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:48:11.998737 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:11.998747 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:11.999735 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:11.999741 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:12.000730 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:12.000742 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:12.001392 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:12.001399 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:12.001554 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:12.002094 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:48:12.002113 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:48:12.005229 (+ 3116us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.243.126:46457: Server oryx/127.3.243.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":251,"threads_started":1}
W20260619 18:48:12.005645 13290 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:48:11.997325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.66:45441 (local address 127.3.243.126:46457)
0619 18:48:11.997538 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0619 18:48:11.997544 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:48:11.997821 (+ 277us) server_negotiation.cc:407] Connection header received
0619 18:48:11.997980 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:48:11.997985 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:48:11.998052 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:48:11.998132 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:48:11.998910 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:11.999603 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:12.000868 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:12.001254 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:12.001358 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:12.005422 (+ 4064us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.243.66:45441: BlockingRecv error: recv got EOF from 127.3.243.66:45441 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 19 18:48:12 dist-test-slave-m9wg krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.67@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:48:12 dist-test-slave-m9wg krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.67@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:48:12.451805 13323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:48:12.443213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:46457 (local address 127.3.243.67:55421)
0619 18:48:12.443729 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0619 18:48:12.443744 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 18:48:12.443864 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:48:12.444235 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:48:12.444240 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:48:12.444655 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:48:12.444828 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:12.444839 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:12.446105 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:12.446109 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:12.447061 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:12.447072 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:12.447673 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:12.447676 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:12.447789 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:12.448346 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:48:12.448363 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:48:12.451493 (+ 3130us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.243.126:46457: Server oryx/127.3.243.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":291,"threads_started":1}
W20260619 18:48:12.451932 13290 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:48:12.443385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.67:55421 (local address 127.3.243.126:46457)
0619 18:48:12.443590 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 18:48:12.443595 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:48:12.443830 (+ 235us) server_negotiation.cc:407] Connection header received
0619 18:48:12.443992 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:48:12.443996 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:48:12.444068 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:48:12.444198 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:48:12.445066 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:12.445961 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:12.447166 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:12.447565 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:12.447653 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:12.451687 (+ 4034us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.243.67:55421: BlockingRecv error: recv got EOF from 127.3.243.67:55421 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Jun 19 18:48:12 dist-test-slave-m9wg krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.65@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:48:12 dist-test-slave-m9wg krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.65@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:48:12.610208 13324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:48:12.601817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:46457 (local address 127.3.243.65:52483)
0619 18:48:12.602294 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0619 18:48:12.602310 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 18:48:12.602420 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:48:12.602692 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:48:12.602695 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:48:12.603087 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:48:12.603255 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:12.603263 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:12.604204 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:12.604207 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:12.605096 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:12.605108 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:12.605794 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:12.605798 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:12.605916 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:12.606437 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:48:12.606462 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:48:12.609885 (+ 3423us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.243.126:46457: Server oryx/127.3.243.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":267,"threads_started":1}
W20260619 18:48:12.610267 13290 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:48:12.601906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.65:52483 (local address 127.3.243.126:46457)
0619 18:48:12.602063 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 18:48:12.602067 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:48:12.602404 (+ 337us) server_negotiation.cc:407] Connection header received
0619 18:48:12.602520 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:48:12.602523 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:48:12.602586 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:48:12.602665 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:48:12.603375 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:12.604077 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:12.605234 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:12.605676 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:12.605769 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:12.610059 (+ 4290us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.243.65:52483: BlockingRecv error: recv got EOF from 127.3.243.65:52483 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 19 18:48:13 dist-test-slave-m9wg krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.66@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:48:13 dist-test-slave-m9wg krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.66@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:48:13.015254 13325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:48:13.006585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:46457 (local address 127.3.243.66:44133)
0619 18:48:13.007083 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0619 18:48:13.007099 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 18:48:13.007219 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:48:13.007520 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:48:13.007523 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:48:13.007962 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:48:13.008185 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:13.008198 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:13.009489 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:13.009495 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:13.010426 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:13.010439 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:13.011073 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:13.011077 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:13.011194 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:13.011711 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:48:13.011729 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:48:13.015036 (+ 3307us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.243.126:46457: Server oryx/127.3.243.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":264,"threads_started":1}
W20260619 18:48:13.015434 13290 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:48:13.006683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.66:44133 (local address 127.3.243.126:46457)
0619 18:48:13.006878 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 18:48:13.006883 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:48:13.007177 (+ 294us) server_negotiation.cc:407] Connection header received
0619 18:48:13.007330 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:48:13.007335 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:48:13.007403 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:48:13.007489 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:48:13.008344 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:13.009354 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:13.010548 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:13.010962 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:13.011052 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:13.015212 (+ 4160us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.243.66:44133: BlockingRecv error: recv got EOF from 127.3.243.66:44133 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Jun 19 18:48:13 dist-test-slave-m9wg krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.67@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:48:13 dist-test-slave-m9wg krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.67@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:48:13.461321 13326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:48:13.452675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:46457 (local address 127.3.243.67:36271)
0619 18:48:13.453132 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0619 18:48:13.453147 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 18:48:13.453271 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:48:13.453583 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:48:13.453588 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:48:13.453992 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:48:13.454176 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:13.454186 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:13.455161 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:13.455166 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:13.456097 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:13.456112 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:13.457094 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:13.457100 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:13.457276 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:13.457809 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:48:13.457825 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:48:13.461099 (+ 3274us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.243.126:46457: Server oryx/127.3.243.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":249,"threads_started":1}
W20260619 18:48:13.461496 13290 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:48:13.452758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.67:36271 (local address 127.3.243.126:46457)
0619 18:48:13.452968 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0619 18:48:13.452975 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:48:13.453219 (+ 244us) server_negotiation.cc:407] Connection header received
0619 18:48:13.453385 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:48:13.453390 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:48:13.453460 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:48:13.453537 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:48:13.454310 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:13.455034 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:13.456317 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:13.456913 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:13.457031 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:13.461290 (+ 4259us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.243.67:36271: BlockingRecv error: recv got EOF from 127.3.243.67:36271 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 19 18:48:13 dist-test-slave-m9wg krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.65@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:48:13 dist-test-slave-m9wg krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.65@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:48:13.619748 13327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:48:13.611107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:46457 (local address 127.3.243.65:57193)
0619 18:48:13.611634 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0619 18:48:13.611649 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 18:48:13.611758 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:48:13.612053 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:48:13.612056 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:48:13.612471 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:48:13.612638 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:13.612646 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:13.613745 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:13.613751 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:13.614732 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:13.614746 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:13.615515 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:13.615519 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:13.615649 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:13.616217 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:48:13.616233 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:48:13.619446 (+ 3213us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.243.126:46457: Server oryx/127.3.243.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":272,"threads_started":1}
W20260619 18:48:13.620038 13290 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:48:13.611164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.65:57193 (local address 127.3.243.126:46457)
0619 18:48:13.611316 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 18:48:13.611320 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:48:13.611719 (+ 399us) server_negotiation.cc:407] Connection header received
0619 18:48:13.611866 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:48:13.611870 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:48:13.611931 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:48:13.612007 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:48:13.612756 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:13.613613 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:13.614856 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:13.615345 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:13.615436 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:13.619676 (+ 4240us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.243.65:57193: BlockingRecv error: recv got EOF from 127.3.243.65:57193 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Jun 19 18:48:14 dist-test-slave-m9wg krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.66@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:48:14 dist-test-slave-m9wg krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.66@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:48:14.024767 13328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:48:14.016211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:46457 (local address 127.3.243.66:49199)
0619 18:48:14.016723 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0619 18:48:14.016738 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 18:48:14.016861 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:48:14.017163 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:48:14.017167 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:48:14.017578 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:48:14.017746 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:14.017755 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:14.018743 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:14.018747 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:14.019804 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:14.019816 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:14.020661 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:14.020668 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:14.020831 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:14.021374 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:48:14.021391 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:48:14.024442 (+ 3051us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.243.126:46457: Server oryx/127.3.243.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":265,"threads_started":1}
W20260619 18:48:14.024873 13290 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:48:14.016310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.66:49199 (local address 127.3.243.126:46457)
0619 18:48:14.016510 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 18:48:14.016516 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:48:14.016812 (+ 296us) server_negotiation.cc:407] Connection header received
0619 18:48:14.016980 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:48:14.016984 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:48:14.017042 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:48:14.017119 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:48:14.017877 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:14.018614 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:14.019928 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:14.020485 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:14.020613 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:14.024636 (+ 4023us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.243.66:49199: BlockingRecv error: recv got EOF from 127.3.243.66:49199 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Jun 19 18:48:14 dist-test-slave-m9wg krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.67@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:48:14 dist-test-slave-m9wg krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.67@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:48:14.471019 13329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:48:14.462230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:46457 (local address 127.3.243.67:55601)
0619 18:48:14.462727 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0619 18:48:14.462742 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 18:48:14.462916 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:48:14.463211 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:48:14.463215 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:48:14.463592 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:48:14.463758 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:14.463766 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:14.464782 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:14.464786 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:14.465781 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:14.465797 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:14.466594 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:14.466601 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:14.466795 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:14.467441 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:48:14.467464 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:48:14.470744 (+ 3280us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.243.126:46457: Server oryx/127.3.243.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":246,"threads_started":1}
W20260619 18:48:14.471127 13290 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:48:14.462302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.67:55601 (local address 127.3.243.126:46457)
0619 18:48:14.462521 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0619 18:48:14.462527 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:48:14.462818 (+ 291us) server_negotiation.cc:407] Connection header received
0619 18:48:14.463016 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:48:14.463020 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:48:14.463106 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:48:14.463181 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:48:14.463894 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:14.464648 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:14.465910 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:14.466422 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:14.466544 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:14.470916 (+ 4372us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.243.67:55601: BlockingRecv error: recv got EOF from 127.3.243.67:55601 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Jun 19 18:48:14 dist-test-slave-m9wg krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.65@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:48:14 dist-test-slave-m9wg krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.65@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:48:14.629005 13330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:48:14.620698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:46457 (local address 127.3.243.65:49091)
0619 18:48:14.621168 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0619 18:48:14.621184 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 18:48:14.621310 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:48:14.621610 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:48:14.621614 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:48:14.622076 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:48:14.622228 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:14.622238 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:14.623186 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:14.623190 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:14.624082 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:14.624093 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:14.624769 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:14.624774 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:14.624936 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:14.625476 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:48:14.625495 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:48:14.628682 (+ 3187us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.243.126:46457: Server oryx/127.3.243.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":287,"threads_started":1}
W20260619 18:48:14.629148 13290 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:48:14.620730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.65:49091 (local address 127.3.243.126:46457)
0619 18:48:14.620916 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 18:48:14.620921 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:48:14.621259 (+ 338us) server_negotiation.cc:407] Connection header received
0619 18:48:14.621425 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:48:14.621430 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:48:14.621493 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:48:14.621566 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:48:14.622361 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:14.623058 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:14.624225 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:14.624637 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:14.624744 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:14.628912 (+ 4168us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.243.65:49091: BlockingRecv error: recv got EOF from 127.3.243.65:49091 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 19 18:48:15 dist-test-slave-m9wg krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.66@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:48:15 dist-test-slave-m9wg krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.66@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:48:15.034166 13331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:48:15.025643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:46457 (local address 127.3.243.66:48675)
0619 18:48:15.026078 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0619 18:48:15.026092 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 18:48:15.026205 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:48:15.026445 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:48:15.026448 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:48:15.026860 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:48:15.027007 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:15.027017 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:15.028064 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:15.028068 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:15.029045 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:15.029058 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:15.029671 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:15.029675 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:15.029788 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:15.030335 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:48:15.030352 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:48:15.033942 (+ 3590us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.243.126:46457: Server oryx/127.3.243.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":230,"threads_started":1}
W20260619 18:48:15.034300 13290 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:48:15.025771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.66:48675 (local address 127.3.243.126:46457)
0619 18:48:15.025938 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 18:48:15.025942 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:48:15.026153 (+ 211us) server_negotiation.cc:407] Connection header received
0619 18:48:15.026284 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:48:15.026287 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:48:15.026344 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:48:15.026416 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:48:15.027111 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:15.027941 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:15.029153 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:15.029568 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:15.029657 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:15.034112 (+ 4455us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.243.66:48675: BlockingRecv error: recv got EOF from 127.3.243.66:48675 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Jun 19 18:48:15 dist-test-slave-m9wg krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.67@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:48:15 dist-test-slave-m9wg krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.67@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:48:15.480078 13332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:48:15.471922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:46457 (local address 127.3.243.67:45519)
0619 18:48:15.472376 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0619 18:48:15.472391 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 18:48:15.472502 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:48:15.472759 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:48:15.472761 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:48:15.473140 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:48:15.473295 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:15.473305 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:15.474336 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:15.474340 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:15.475255 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:15.475266 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:15.475979 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:15.475984 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:15.476119 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:15.476695 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:48:15.476714 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:48:15.479858 (+ 3144us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.243.126:46457: Server oryx/127.3.243.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":238,"threads_started":1}
W20260619 18:48:15.480234 13290 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:48:15.472036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.67:45519 (local address 127.3.243.126:46457)
0619 18:48:15.472284 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0619 18:48:15.472290 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:48:15.472457 (+ 167us) server_negotiation.cc:407] Connection header received
0619 18:48:15.472593 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:48:15.472596 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:48:15.472656 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:48:15.472730 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:48:15.473444 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:15.474204 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:15.475393 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:15.475822 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:15.475916 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:15.480017 (+ 4101us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.243.67:45519: BlockingRecv error: recv got EOF from 127.3.243.67:45519 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Jun 19 18:48:15 dist-test-slave-m9wg krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.65@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:48:15 dist-test-slave-m9wg krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.65@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:48:15.639335 13333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:48:15.630516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:46457 (local address 127.3.243.65:54591)
0619 18:48:15.631096 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0619 18:48:15.631113 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 18:48:15.631239 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:48:15.631547 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:48:15.631552 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:48:15.631946 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:48:15.632139 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:15.632175 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:15.633247 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:15.633252 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:15.634210 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:15.634223 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:15.634915 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:15.634921 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:15.635098 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:15.635643 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:48:15.635661 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:48:15.639083 (+ 3422us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.243.126:46457: Server oryx/127.3.243.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":309,"threads_started":1}
W20260619 18:48:15.639489 13290 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:48:15.630551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.65:54591 (local address 127.3.243.126:46457)
0619 18:48:15.630804 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0619 18:48:15.630810 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:48:15.631195 (+ 385us) server_negotiation.cc:407] Connection header received
0619 18:48:15.631349 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:48:15.631354 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:48:15.631424 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:48:15.631510 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:48:15.632309 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:15.633092 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:15.634346 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:15.634780 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:15.634890 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:15.639264 (+ 4374us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.243.65:54591: BlockingRecv error: recv got EOF from 127.3.243.65:54591 (error 108)
Metrics: {"server-negotiator.queue_time_us":130}
Jun 19 18:48:16 dist-test-slave-m9wg krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.66@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:48:16 dist-test-slave-m9wg krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.66@KRBTEST.COM for oryx/127.3.243.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:48:16.046478 13334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:48:16.035183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:46457 (local address 127.3.243.66:57989)
0619 18:48:16.035599 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0619 18:48:16.035614 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 18:48:16.035720 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:48:16.035979 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:48:16.035982 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:48:16.036370 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:48:16.036575 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:16.036588 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:16.037577 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:16.037581 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:16.038477 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:16.038488 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:16.039070 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:16.039074 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:16.039188 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:16.042818 (+ 3630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:48:16.042836 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:48:16.046230 (+ 3394us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.243.126:46457: Server oryx/127.3.243.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":191,"threads_started":1}
W20260619 18:48:16.046597 13290 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:48:16.035238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.66:57989 (local address 127.3.243.126:46457)
0619 18:48:16.035449 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0619 18:48:16.035456 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:48:16.035681 (+ 225us) server_negotiation.cc:407] Connection header received
0619 18:48:16.035815 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:48:16.035819 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:48:16.035879 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:48:16.035952 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:48:16.036702 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:16.037455 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:16.038589 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:16.038968 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:16.039056 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:16.046391 (+ 7335us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.243.66:57989: BlockingRecv error: recv got EOF from 127.3.243.66:57989 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 18:48:16.239812 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 12873
I20260619 18:48:16.250946 13336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:48:16.240732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40352 (local address 127.3.243.65:41923)
0619 18:48:16.241274 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0619 18:48:16.241280 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:48:16.241306 (+ 26us) server_negotiation.cc:407] Connection header received
0619 18:48:16.241366 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:48:16.241374 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:48:16.241608 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:48:16.241765 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:48:16.243058 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:16.244051 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:16.245170 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:16.245709 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:16.245803 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:16.247968 (+ 2165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:48:16.247980 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:48:16.247999 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:48:16.248054 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:48:16.250475 (+ 2421us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.3.243.65@KRBTEST.COM not found in keytab (ticket kvno 2)
0619 18:48:16.250736 (+ 261us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40352: Request ticket server kudu/127.3.243.65@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":275,"threads_started":1}
W20260619 18:48:16.251035 13336 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40352: Request ticket server kudu/127.3.243.65@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260619 18:48:16.251068 4045 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 12873: Not authorized: Client connection negotiation failed: client connection to 127.3.243.65:41923: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.3.243.65@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260619 18:48:16.251283 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 12873
I20260619 18:48:16.264544 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 13024
I20260619 18:48:16.275867 13337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:48:16.265508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58718 (local address 127.3.243.66:46659)
0619 18:48:16.266124 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0619 18:48:16.266131 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:48:16.266158 (+ 27us) server_negotiation.cc:407] Connection header received
0619 18:48:16.266214 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:48:16.266220 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:48:16.266442 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:48:16.266595 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:48:16.268096 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:16.269179 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:16.270271 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:16.270804 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:16.270900 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:16.273116 (+ 2216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:48:16.273125 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:48:16.273137 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:48:16.273170 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:48:16.275361 (+ 2191us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.3.243.66@KRBTEST.COM not found in keytab (ticket kvno 2)
0619 18:48:16.275615 (+ 254us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58718: Request ticket server kudu/127.3.243.66@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":313,"threads_started":1}
W20260619 18:48:16.275941 4045 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 13024: Not authorized: Client connection negotiation failed: client connection to 127.3.243.66:46659: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.3.243.66@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260619 18:48:16.275965 13337 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58718: Request ticket server kudu/127.3.243.66@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260619 18:48:16.276059 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 13024
I20260619 18:48:16.289188 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 13157
I20260619 18:48:16.300854 13338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:48:16.290201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56680 (local address 127.3.243.67:40123)
0619 18:48:16.290751 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0619 18:48:16.290758 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:48:16.290783 (+ 25us) server_negotiation.cc:407] Connection header received
0619 18:48:16.290843 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:48:16.290849 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:48:16.291089 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:48:16.291246 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:48:16.292885 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:16.293906 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:16.295007 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:16.295549 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:16.295645 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:16.297996 (+ 2351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:48:16.298004 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:48:16.298019 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:48:16.298059 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:48:16.300356 (+ 2297us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.3.243.67@KRBTEST.COM not found in keytab (ticket kvno 2)
0619 18:48:16.300625 (+ 269us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56680: Request ticket server kudu/127.3.243.67@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":274,"threads_started":1}
W20260619 18:48:16.300910 4045 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 13157: Not authorized: Client connection negotiation failed: client connection to 127.3.243.67:40123: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.3.243.67@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260619 18:48:16.300940 13338 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56680: Request ticket server kudu/127.3.243.67@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260619 18:48:16.301059 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 13157
I20260619 18:48:16.313953 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 12804
I20260619 18:48:16.315078 12866 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:48:16.431588 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 12804
2026-06-19T18:48:16Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (20644 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-19T18:48:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:48:16Z Disabled control of system clock
I20260619 18:48:16.498077 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894810715282-4045-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894810715282-4045-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894810715282-4045-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.126:33823
--webserver_interface=127.3.243.126
--webserver_port=0
--builtin_ntp_servers=127.3.243.84:37399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.126:33823
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 18:48:16.804509 13343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:48:16.804775 13343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:48:16.804859 13343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:48:16.811731 13343 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:48:16.811873 13343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:48:16.811918 13343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:48:16.811970 13343 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:48:16.812012 13343 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:48:16.820914 13343 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:37399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894810715282-4045-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894810715282-4045-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.243.126:33823
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.126:33823
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.243.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894810715282-4045-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:48:16.822074 13343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:48:16.823799 13343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:48:16.834769 13348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:48:16.834820 13349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:48:16.834791 13351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:48:16.835680 13343 server_base.cc:1034] running on GCE node
I20260619 18:48:16.836499 13343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:48:16.837782 13343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:48:16.838995 13343 hybrid_clock.cc:648] HybridClock initialized: now 1781894896838931 us; error 44 us; skew 500 ppm
I20260619 18:48:16.839334 13343 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 18:48:16.841512 13343 webserver.cc:460] Webserver started at http://127.3.243.126:37735/ using document root <none> and password file <none>
I20260619 18:48:16.842151 13343 fs_manager.cc:359] Metadata directory not provided
I20260619 18:48:16.842242 13343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:48:16.842541 13343 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:48:16.845434 13343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894810715282-4045-0/minicluster-data/master-0/data/instance:
uuid: "97cc12d19307473a93470d2775019d37"
format_stamp: "Formatted at 2026-06-19 18:48:16 on dist-test-slave-m9wg"
server_key: "6b8e487664942383a9cc4aad28bacd5d"
server_key_iv: "7f77ebf49b6f6e6803e959e1988c2fd9"
server_key_version: "encryptionkey@0"
I20260619 18:48:16.846167 13343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894810715282-4045-0/minicluster-data/master-0/wal/instance:
uuid: "97cc12d19307473a93470d2775019d37"
format_stamp: "Formatted at 2026-06-19 18:48:16 on dist-test-slave-m9wg"
server_key: "6b8e487664942383a9cc4aad28bacd5d"
server_key_iv: "7f77ebf49b6f6e6803e959e1988c2fd9"
server_key_version: "encryptionkey@0"
I20260619 18:48:16.850943 13343 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260619 18:48:16.854151 13357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:48:16.855368 13343 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 18:48:16.855510 13343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894810715282-4045-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894810715282-4045-0/minicluster-data/master-0/wal
uuid: "97cc12d19307473a93470d2775019d37"
format_stamp: "Formatted at 2026-06-19 18:48:16 on dist-test-slave-m9wg"
server_key: "6b8e487664942383a9cc4aad28bacd5d"
server_key_iv: "7f77ebf49b6f6e6803e959e1988c2fd9"
server_key_version: "encryptionkey@0"
I20260619 18:48:16.855688 13343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894810715282-4045-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894810715282-4045-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894810715282-4045-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:48:16.871315 13343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:48:16.872046 13343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:48:16.872344 13343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:48:16.888726 13343 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.126:33823
I20260619 18:48:16.888747 13408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.126:33823 every 8 connection(s)
I20260619 18:48:16.889884 13343 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
I20260619 18:48:16.893239 13409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:48:16.894258 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 13343
I20260619 18:48:16.894418 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894810715282-4045-0/minicluster-data/master-0/wal/instance
I20260619 18:48:16.895013 4045 external_mini_cluster.cc:1428] Setting key 41a4625c4ebe09a983e660870290e777
I20260619 18:48:16.906512 13409 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97cc12d19307473a93470d2775019d37: Bootstrap starting.
I20260619 18:48:16.908847 13412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:48:16.896924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38948 (local address 127.3.243.126:33823)
0619 18:48:16.897932 (+ 1008us) server_negotiation.cc:206] Beginning negotiation
0619 18:48:16.897946 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 18:48:16.897995 (+ 49us) server_negotiation.cc:407] Connection header received
0619 18:48:16.899536 (+ 1541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:48:16.899563 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:48:16.900137 (+ 574us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:48:16.900569 (+ 432us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:48:16.901027 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:16.902792 (+ 1765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:16.904063 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:16.905069 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:16.905234 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:16.906264 (+ 1030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:48:16.906275 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:48:16.906291 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:48:16.906934 (+ 643us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:48:16.907024 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:48:16.907048 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:48:16.907160 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 18:48:16.907350 (+ 190us) server_negotiation.cc:299] Negotiation successful
0619 18:48:16.907630 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":300,"threads_started":1}
I20260619 18:48:16.909957 13409 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 97cc12d19307473a93470d2775019d37: Neither blocks nor log segments found. Creating new log.
I20260619 18:48:16.910917 13409 log.cc:824] T 00000000000000000000000000000000 P 97cc12d19307473a93470d2775019d37: Log is configured to *not* fsync() on all Append() calls
I20260619 18:48:16.913533 13409 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97cc12d19307473a93470d2775019d37: No bootstrap required, opened a new log
I20260619 18:48:16.920133 13409 raft_consensus.cc:348] T 00000000000000000000000000000000 P 97cc12d19307473a93470d2775019d37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97cc12d19307473a93470d2775019d37" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 33823 } }
I20260619 18:48:16.920540 13409 raft_consensus.cc:374] T 00000000000000000000000000000000 P 97cc12d19307473a93470d2775019d37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:48:16.920630 13409 raft_consensus.cc:729] T 00000000000000000000000000000000 P 97cc12d19307473a93470d2775019d37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97cc12d19307473a93470d2775019d37, State: Initialized, Role: FOLLOWER
I20260619 18:48:16.921123 13409 consensus_queue.cc:260] T 00000000000000000000000000000000 P 97cc12d19307473a93470d2775019d37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97cc12d19307473a93470d2775019d37" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 33823 } }
I20260619 18:48:16.921309 13409 raft_consensus.cc:388] T 00000000000000000000000000000000 P 97cc12d19307473a93470d2775019d37 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:48:16.921433 13409 raft_consensus.cc:482] T 00000000000000000000000000000000 P 97cc12d19307473a93470d2775019d37 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:48:16.921566 13409 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 97cc12d19307473a93470d2775019d37 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:48:16.923465 13409 raft_consensus.cc:504] T 00000000000000000000000000000000 P 97cc12d19307473a93470d2775019d37 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97cc12d19307473a93470d2775019d37" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 33823 } }
I20260619 18:48:16.923888 13409 leader_election.cc:302] T 00000000000000000000000000000000 P 97cc12d19307473a93470d2775019d37 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 97cc12d19307473a93470d2775019d37; no voters:
I20260619 18:48:16.924469 13409 leader_election.cc:288] T 00000000000000000000000000000000 P 97cc12d19307473a93470d2775019d37 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:48:16.924593 13414 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 97cc12d19307473a93470d2775019d37 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:48:16.925230 13414 raft_consensus.cc:686] T 00000000000000000000000000000000 P 97cc12d19307473a93470d2775019d37 [term 1 LEADER]: Becoming Leader. State: Replica: 97cc12d19307473a93470d2775019d37, State: Running, Role: LEADER
I20260619 18:48:16.925729 13414 consensus_queue.cc:237] T 00000000000000000000000000000000 P 97cc12d19307473a93470d2775019d37 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97cc12d19307473a93470d2775019d37" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 33823 } }
I20260619 18:48:16.926308 13409 sys_catalog.cc:564] T 00000000000000000000000000000000 P 97cc12d19307473a93470d2775019d37 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:48:16.928478 13416 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97cc12d19307473a93470d2775019d37 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 97cc12d19307473a93470d2775019d37. Latest consensus state: current_term: 1 leader_uuid: "97cc12d19307473a93470d2775019d37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97cc12d19307473a93470d2775019d37" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 33823 } } }
I20260619 18:48:16.928548 13415 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97cc12d19307473a93470d2775019d37 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "97cc12d19307473a93470d2775019d37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97cc12d19307473a93470d2775019d37" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 33823 } } }
I20260619 18:48:16.928830 13416 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97cc12d19307473a93470d2775019d37 [sys.catalog]: This master's current role is: LEADER
I20260619 18:48:16.928881 13415 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97cc12d19307473a93470d2775019d37 [sys.catalog]: This master's current role is: LEADER
I20260619 18:48:16.933437 13421 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:48:16.938385 13421 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:48:16.946133 13421 catalog_manager.cc:1347] Generated new cluster ID: ccc1e07aa1954d7da8d72df7beb2aa6e
I20260619 18:48:16.946228 13421 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:48:16.952816 13421 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:48:16.954417 13421 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:48:16.970789 13421 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 97cc12d19307473a93470d2775019d37: Generated new TSK 0
I20260619 18:48:16.971715 13421 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:48:16.976101 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894810715282-4045-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894810715282-4045-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894810715282-4045-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894810715282-4045-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.65:0
--local_ip_for_outbound_sockets=127.3.243.65
--webserver_interface=127.3.243.65
--webserver_port=0
--tserver_master_addrs=127.3.243.126:33823
--builtin_ntp_servers=127.3.243.84:37399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260619 18:48:17.279542 13433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:48:17.279815 13433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:48:17.279930 13433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:48:17.286954 13433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:48:17.287055 13433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:48:17.287176 13433 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.65
I20260619 18:48:17.295825 13433 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:37399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894810715282-4045-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894810715282-4045-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894810715282-4045-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.243.65
--webserver_port=0
--tserver_master_addrs=127.3.243.126:33823
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.65
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894810715282-4045-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:48:17.297040 13433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:48:17.298732 13433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:48:17.310518 13439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:48:17.310551 13441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:48:17.310490 13438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:48:17.311156 13433 server_base.cc:1034] running on GCE node
I20260619 18:48:17.311721 13433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:48:17.312619 13433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:48:17.313822 13433 hybrid_clock.cc:648] HybridClock initialized: now 1781894897313774 us; error 34 us; skew 500 ppm
I20260619 18:48:17.314106 13433 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 18:48:17.316140 13433 webserver.cc:460] Webserver started at http://127.3.243.65:42307/ using document root <none> and password file <none>
I20260619 18:48:17.316859 13433 fs_manager.cc:359] Metadata directory not provided
I20260619 18:48:17.316956 13433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:48:17.317263 13433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:48:17.319936 13433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894810715282-4045-0/minicluster-data/ts-0/data/instance:
uuid: "78becbde2bbe4921ad944cbf9fa70c23"
format_stamp: "Formatted at 2026-06-19 18:48:17 on dist-test-slave-m9wg"
server_key: "609f79db3905d1afe2bdce9614dcd6f5"
server_key_iv: "d952581bb46593b91c7e02b81747fb6d"
server_key_version: "encryptionkey@0"
I20260619 18:48:17.320719 13433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894810715282-4045-0/minicluster-data/ts-0/wal/instance:
uuid: "78becbde2bbe4921ad944cbf9fa70c23"
format_stamp: "Formatted at 2026-06-19 18:48:17 on dist-test-slave-m9wg"
server_key: "609f79db3905d1afe2bdce9614dcd6f5"
server_key_iv: "d952581bb46593b91c7e02b81747fb6d"
server_key_version: "encryptionkey@0"
I20260619 18:48:17.325459 13433 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260619 18:48:17.328773 13447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:48:17.329991 13433 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s
I20260619 18:48:17.330133 13433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894810715282-4045-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894810715282-4045-0/minicluster-data/ts-0/wal
uuid: "78becbde2bbe4921ad944cbf9fa70c23"
format_stamp: "Formatted at 2026-06-19 18:48:17 on dist-test-slave-m9wg"
server_key: "609f79db3905d1afe2bdce9614dcd6f5"
server_key_iv: "d952581bb46593b91c7e02b81747fb6d"
server_key_version: "encryptionkey@0"
I20260619 18:48:17.330327 13433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894810715282-4045-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894810715282-4045-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894810715282-4045-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:48:17.344111 13433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:48:17.344929 13433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:48:17.345199 13433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:48:17.346117 13433 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:48:17.347663 13433 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:48:17.347759 13433 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:48:17.347860 13433 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:48:17.347929 13433 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:48:17.377768 13433 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.65:34529
I20260619 18:48:17.377808 13559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.65:34529 every 8 connection(s)
I20260619 18:48:17.379019 13433 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894810715282-4045-0/minicluster-data/ts-0/data/info.pb
I20260619 18:48:17.387887 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 13433
I20260619 18:48:17.388072 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894810715282-4045-0/minicluster-data/ts-0/wal/instance
I20260619 18:48:17.388713 4045 external_mini_cluster.cc:1428] Setting key 4ab553f1132ffb85c897e4bc3ef6fcdf
I20260619 18:48:17.390625 13412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:48:17.381530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.65:46383 (local address 127.3.243.126:33823)
0619 18:48:17.381733 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 18:48:17.381737 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:48:17.383019 (+ 1282us) server_negotiation.cc:407] Connection header received
0619 18:48:17.384725 (+ 1706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:48:17.384730 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:48:17.384795 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:48:17.384881 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:48:17.385888 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:17.386577 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:17.388097 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:17.388564 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:17.388677 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:17.389953 (+ 1276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:48:17.389958 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:48:17.389961 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:48:17.389978 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:48:17.390007 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:48:17.390018 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:48:17.390089 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 18:48:17.390293 (+ 204us) server_negotiation.cc:299] Negotiation successful
0619 18:48:17.390376 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260619 18:48:17.391340 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894810715282-4045-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894810715282-4045-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894810715282-4045-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894810715282-4045-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.66:0
--local_ip_for_outbound_sockets=127.3.243.66
--webserver_interface=127.3.243.66
--webserver_port=0
--tserver_master_addrs=127.3.243.126:33823
--builtin_ntp_servers=127.3.243.84:37399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260619 18:48:17.391639 13562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:48:17.381987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:33823 (local address 127.3.243.65:46383)
0619 18:48:17.382883 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0619 18:48:17.382919 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 18:48:17.384495 (+ 1576us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:48:17.385063 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:48:17.385071 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:48:17.385157 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:48:17.385714 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:17.385731 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:17.386705 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:17.386709 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:17.387967 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:17.387982 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:17.388783 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:17.388791 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:17.389002 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:17.389630 (+ 628us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:48:17.389632 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:48:17.389669 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:48:17.389738 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:48:17.390101 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:48:17.390112 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:48:17.390120 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 18:48:17.390268 (+ 148us) client_negotiation.cc:240] Negotiation successful
0619 18:48:17.390606 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":259,"threads_started":1}
I20260619 18:48:17.393126 13560 heartbeater.cc:347] Connected to a master server at 127.3.243.126:33823
I20260619 18:48:17.393388 13560 heartbeater.cc:464] Registering TS with master...
I20260619 18:48:17.394089 13560 heartbeater.cc:511] Master 127.3.243.126:33823 requested a full tablet report, sending...
I20260619 18:48:17.396181 13374 ts_manager.cc:194] Registered new tserver with Master: 78becbde2bbe4921ad944cbf9fa70c23 (127.3.243.65:34529)
I20260619 18:48:17.398073 13374 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.243.65:46383
W20260619 18:48:17.698712 13563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:48:17.699023 13563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:48:17.699148 13563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:48:17.706562 13563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:48:17.706669 13563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:48:17.706813 13563 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.66
I20260619 18:48:17.715692 13563 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:37399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894810715282-4045-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894810715282-4045-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894810715282-4045-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.243.66
--webserver_port=0
--tserver_master_addrs=127.3.243.126:33823
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.66
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894810715282-4045-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:48:17.716989 13563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:48:17.718668 13563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:48:17.731103 13568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:48:17.731056 13571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:48:17.731030 13569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:48:17.731593 13563 server_base.cc:1034] running on GCE node
I20260619 18:48:17.732266 13563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:48:17.733196 13563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:48:17.734395 13563 hybrid_clock.cc:648] HybridClock initialized: now 1781894897734338 us; error 51 us; skew 500 ppm
I20260619 18:48:17.734686 13563 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 18:48:17.736937 13563 webserver.cc:460] Webserver started at http://127.3.243.66:41703/ using document root <none> and password file <none>
I20260619 18:48:17.737584 13563 fs_manager.cc:359] Metadata directory not provided
I20260619 18:48:17.737680 13563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:48:17.737986 13563 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:48:17.740783 13563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894810715282-4045-0/minicluster-data/ts-1/data/instance:
uuid: "a655864a5a624358ae9e737a6790d7ed"
format_stamp: "Formatted at 2026-06-19 18:48:17 on dist-test-slave-m9wg"
server_key: "7c6085103273d1cf9a4729e8a5e7fad2"
server_key_iv: "8e4a7d8f1dd78a82ebb2067522d4f2ef"
server_key_version: "encryptionkey@0"
I20260619 18:48:17.741549 13563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894810715282-4045-0/minicluster-data/ts-1/wal/instance:
uuid: "a655864a5a624358ae9e737a6790d7ed"
format_stamp: "Formatted at 2026-06-19 18:48:17 on dist-test-slave-m9wg"
server_key: "7c6085103273d1cf9a4729e8a5e7fad2"
server_key_iv: "8e4a7d8f1dd78a82ebb2067522d4f2ef"
server_key_version: "encryptionkey@0"
I20260619 18:48:17.746426 13563 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260619 18:48:17.749835 13577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:48:17.751235 13563 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 18:48:17.751386 13563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894810715282-4045-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894810715282-4045-0/minicluster-data/ts-1/wal
uuid: "a655864a5a624358ae9e737a6790d7ed"
format_stamp: "Formatted at 2026-06-19 18:48:17 on dist-test-slave-m9wg"
server_key: "7c6085103273d1cf9a4729e8a5e7fad2"
server_key_iv: "8e4a7d8f1dd78a82ebb2067522d4f2ef"
server_key_version: "encryptionkey@0"
I20260619 18:48:17.751577 13563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894810715282-4045-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894810715282-4045-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894810715282-4045-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:48:17.772125 13563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:48:17.772981 13563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:48:17.773258 13563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:48:17.774151 13563 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:48:17.775523 13563 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:48:17.775614 13563 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:48:17.775717 13563 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:48:17.775806 13563 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:48:17.804361 13563 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.66:38659
I20260619 18:48:17.804427 13689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.66:38659 every 8 connection(s)
I20260619 18:48:17.805648 13563 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894810715282-4045-0/minicluster-data/ts-1/data/info.pb
I20260619 18:48:17.811957 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 13563
I20260619 18:48:17.812099 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894810715282-4045-0/minicluster-data/ts-1/wal/instance
I20260619 18:48:17.812570 4045 external_mini_cluster.cc:1428] Setting key 564aaf3a1859fbe5b06d03c28fcdd0f8
I20260619 18:48:17.815321 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894810715282-4045-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894810715282-4045-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894810715282-4045-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894810715282-4045-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.67:0
--local_ip_for_outbound_sockets=127.3.243.67
--webserver_interface=127.3.243.67
--webserver_port=0
--tserver_master_addrs=127.3.243.126:33823
--builtin_ntp_servers=127.3.243.84:37399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260619 18:48:17.817960 13412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:48:17.807992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.66:34805 (local address 127.3.243.126:33823)
0619 18:48:17.808263 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0619 18:48:17.808269 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:48:17.809650 (+ 1381us) server_negotiation.cc:407] Connection header received
0619 18:48:17.811290 (+ 1640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:48:17.811294 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:48:17.811359 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:48:17.811435 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:48:17.812513 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:17.813449 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:17.815337 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:17.815907 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:17.816029 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:17.817251 (+ 1222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:48:17.817257 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:48:17.817261 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:48:17.817280 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:48:17.817316 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:48:17.817329 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:48:17.817403 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 18:48:17.817579 (+ 176us) server_negotiation.cc:299] Negotiation successful
0619 18:48:17.817667 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 18:48:17.818800 13692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:48:17.808588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:33823 (local address 127.3.243.66:34805)
0619 18:48:17.809497 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0619 18:48:17.809535 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 18:48:17.811048 (+ 1513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:48:17.811633 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:48:17.811642 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:48:17.811727 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:48:17.812345 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:17.812372 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:17.813620 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:17.813625 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:17.815190 (+ 1565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:17.815201 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:17.816089 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:17.816094 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:17.816321 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:17.816956 (+ 635us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:48:17.816958 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:48:17.816994 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:48:17.817070 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:48:17.817417 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:48:17.817423 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:48:17.817428 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 18:48:17.817545 (+ 117us) client_negotiation.cc:240] Negotiation successful
0619 18:48:17.817819 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":266,"threads_started":1}
I20260619 18:48:17.820034 13690 heartbeater.cc:347] Connected to a master server at 127.3.243.126:33823
I20260619 18:48:17.820358 13690 heartbeater.cc:464] Registering TS with master...
I20260619 18:48:17.821201 13690 heartbeater.cc:511] Master 127.3.243.126:33823 requested a full tablet report, sending...
I20260619 18:48:17.822649 13374 ts_manager.cc:194] Registered new tserver with Master: a655864a5a624358ae9e737a6790d7ed (127.3.243.66:38659)
I20260619 18:48:17.823495 13374 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.243.66:34805
W20260619 18:48:18.131735 13693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:48:18.132018 13693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:48:18.132141 13693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:48:18.139400 13693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:48:18.139503 13693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:48:18.139631 13693 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.67
I20260619 18:48:18.148486 13693 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:37399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894810715282-4045-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894810715282-4045-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894810715282-4045-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.243.67
--webserver_port=0
--tserver_master_addrs=127.3.243.126:33823
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.67
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894810715282-4045-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:48:18.149669 13693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:48:18.151387 13693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:48:18.163949 13698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:48:18.163968 13699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:48:18.164072 13701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:48:18.164779 13693 server_base.cc:1034] running on GCE node
I20260619 18:48:18.165395 13693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:48:18.166316 13693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:48:18.167506 13693 hybrid_clock.cc:648] HybridClock initialized: now 1781894898167454 us; error 50 us; skew 500 ppm
I20260619 18:48:18.167802 13693 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 18:48:18.170006 13693 webserver.cc:460] Webserver started at http://127.3.243.67:37705/ using document root <none> and password file <none>
I20260619 18:48:18.170677 13693 fs_manager.cc:359] Metadata directory not provided
I20260619 18:48:18.170776 13693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:48:18.171084 13693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:48:18.173813 13693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894810715282-4045-0/minicluster-data/ts-2/data/instance:
uuid: "831edbc1319d4be3b503b3a8aedd67bc"
format_stamp: "Formatted at 2026-06-19 18:48:18 on dist-test-slave-m9wg"
server_key: "79d700bbc6da7ef1ab6b3ecad0e18716"
server_key_iv: "04a54edea540cd3adbec785387810ace"
server_key_version: "encryptionkey@0"
I20260619 18:48:18.174567 13693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894810715282-4045-0/minicluster-data/ts-2/wal/instance:
uuid: "831edbc1319d4be3b503b3a8aedd67bc"
format_stamp: "Formatted at 2026-06-19 18:48:18 on dist-test-slave-m9wg"
server_key: "79d700bbc6da7ef1ab6b3ecad0e18716"
server_key_iv: "04a54edea540cd3adbec785387810ace"
server_key_version: "encryptionkey@0"
I20260619 18:48:18.179389 13693 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.005s
I20260619 18:48:18.182499 13707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:48:18.183692 13693 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 18:48:18.183832 13693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894810715282-4045-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894810715282-4045-0/minicluster-data/ts-2/wal
uuid: "831edbc1319d4be3b503b3a8aedd67bc"
format_stamp: "Formatted at 2026-06-19 18:48:18 on dist-test-slave-m9wg"
server_key: "79d700bbc6da7ef1ab6b3ecad0e18716"
server_key_iv: "04a54edea540cd3adbec785387810ace"
server_key_version: "encryptionkey@0"
I20260619 18:48:18.184023 13693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894810715282-4045-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894810715282-4045-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894810715282-4045-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:48:18.200613 13693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:48:18.201325 13693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:48:18.201572 13693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:48:18.202406 13693 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:48:18.203703 13693 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:48:18.203794 13693 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:48:18.203893 13693 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:48:18.203967 13693 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:48:18.232776 13693 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.67:39617
I20260619 18:48:18.232838 13819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.67:39617 every 8 connection(s)
I20260619 18:48:18.234071 13693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894810715282-4045-0/minicluster-data/ts-2/data/info.pb
I20260619 18:48:18.238337 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 13693
I20260619 18:48:18.238479 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894810715282-4045-0/minicluster-data/ts-2/wal/instance
I20260619 18:48:18.238898 4045 external_mini_cluster.cc:1428] Setting key 53fd2a91ecf054db814114e0facbad3c
I20260619 18:48:18.246104 13412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:48:18.236526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.67:40599 (local address 127.3.243.126:33823)
0619 18:48:18.236735 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0619 18:48:18.236741 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:48:18.238116 (+ 1375us) server_negotiation.cc:407] Connection header received
0619 18:48:18.239881 (+ 1765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:48:18.239886 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:48:18.239966 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:48:18.240058 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:48:18.241341 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:18.242239 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:18.243748 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:18.244123 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:18.244245 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:18.245412 (+ 1167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:48:18.245416 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:48:18.245419 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:48:18.245436 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:48:18.245460 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:48:18.245471 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:48:18.245532 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 18:48:18.245772 (+ 240us) server_negotiation.cc:299] Negotiation successful
0619 18:48:18.245877 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 18:48:18.246999 13822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:48:18.237081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:33823 (local address 127.3.243.67:40599)
0619 18:48:18.237943 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0619 18:48:18.238003 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0619 18:48:18.239628 (+ 1625us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:48:18.240327 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:48:18.240335 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:48:18.240426 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:48:18.241171 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:18.241196 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:18.242386 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:18.242389 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:18.243636 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:18.243652 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:18.244322 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:18.244326 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:18.244514 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:18.245155 (+ 641us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:48:18.245157 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:48:18.245195 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:48:18.245271 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:48:18.245549 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:48:18.245556 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:48:18.245561 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 18:48:18.245699 (+ 138us) client_negotiation.cc:240] Negotiation successful
0619 18:48:18.246016 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":255,"threads_started":1}
I20260619 18:48:18.248201 13820 heartbeater.cc:347] Connected to a master server at 127.3.243.126:33823
I20260619 18:48:18.248474 13820 heartbeater.cc:464] Registering TS with master...
I20260619 18:48:18.249187 13820 heartbeater.cc:511] Master 127.3.243.126:33823 requested a full tablet report, sending...
I20260619 18:48:18.250483 13374 ts_manager.cc:194] Registered new tserver with Master: 831edbc1319d4be3b503b3a8aedd67bc (127.3.243.67:39617)
I20260619 18:48:18.251237 13374 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.243.67:40599
I20260619 18:48:18.253955 4045 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 18:48:18.259672 13412 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:48:18.258101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38960 (local address 127.3.243.126:33823)
0619 18:48:18.258254 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 18:48:18.258258 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:48:18.258752 (+ 494us) server_negotiation.cc:407] Connection header received
0619 18:48:18.258917 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:48:18.258920 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:48:18.258993 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:48:18.259069 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:48:18.259435 (+ 366us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38960: BlockingRecv error: recv got EOF from 127.0.0.1:38960 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 18:48:18.260334 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 13433
I20260619 18:48:18.267536 13829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:48:18.261197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33194 (local address 127.3.243.65:34529)
0619 18:48:18.261670 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0619 18:48:18.261677 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:48:18.261701 (+ 24us) server_negotiation.cc:407] Connection header received
0619 18:48:18.261796 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:48:18.261803 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:48:18.261991 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:48:18.262116 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:48:18.262413 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:18.263400 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:18.264678 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:18.265214 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:18.265323 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:18.266270 (+ 947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:48:18.266278 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:48:18.266289 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:48:18.266861 (+ 572us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:48:18.266912 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:48:18.266932 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:48:18.267013 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 18:48:18.267139 (+ 126us) server_negotiation.cc:299] Negotiation successful
0619 18:48:18.267261 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":252,"threads_started":1}
I20260619 18:48:18.268841 13555 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:48:18.348551 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 13433
I20260619 18:48:18.362852 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 13563
I20260619 18:48:18.370373 13831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:48:18.363703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49466 (local address 127.3.243.66:38659)
0619 18:48:18.364202 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0619 18:48:18.364212 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:48:18.364242 (+ 30us) server_negotiation.cc:407] Connection header received
0619 18:48:18.364318 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:48:18.364325 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:48:18.364540 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:48:18.364683 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:48:18.365077 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:18.366071 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:18.367256 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:18.367806 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:18.367920 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:18.368996 (+ 1076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:48:18.369004 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:48:18.369017 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:48:18.369605 (+ 588us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:48:18.369650 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:48:18.369669 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:48:18.369774 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 18:48:18.369924 (+ 150us) server_negotiation.cc:299] Negotiation successful
0619 18:48:18.370064 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":229,"threads_started":1}
I20260619 18:48:18.371647 13685 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:48:18.451325 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 13563
I20260619 18:48:18.464394 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 13693
I20260619 18:48:18.472476 13833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:48:18.465456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46832 (local address 127.3.243.67:39617)
0619 18:48:18.466094 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0619 18:48:18.466101 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:48:18.466130 (+ 29us) server_negotiation.cc:407] Connection header received
0619 18:48:18.466189 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:48:18.466195 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:48:18.466406 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:48:18.466529 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:48:18.466920 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:18.467914 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:18.469240 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:18.469803 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:18.469909 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:18.470974 (+ 1065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:48:18.470982 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:48:18.470996 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:48:18.471618 (+ 622us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:48:18.471666 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:48:18.471687 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:48:18.471769 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 18:48:18.471927 (+ 158us) server_negotiation.cc:299] Negotiation successful
0619 18:48:18.472054 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":322,"threads_started":1}
I20260619 18:48:18.473719 13815 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:48:18.554563 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 13693
I20260619 18:48:18.567777 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 13343
I20260619 18:48:18.569228 13404 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:48:18.673259 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 13343
2026-06-19T18:48:18Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2237 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-19T18:48:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:48:18Z Disabled control of system clock
I20260619 18:48:18.738241 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894810715282-4045-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894810715282-4045-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894810715282-4045-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.126:43507
--webserver_interface=127.3.243.126
--webserver_port=0
--builtin_ntp_servers=127.3.243.84:33861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.126:43507
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 18:48:19.041365 13839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:48:19.041647 13839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:48:19.041738 13839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:48:19.048769 13839 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:48:19.048866 13839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:48:19.048908 13839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:48:19.048962 13839 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:48:19.049006 13839 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:48:19.057828 13839 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:33861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894810715282-4045-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894810715282-4045-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.243.126:43507
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.126:43507
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.243.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894810715282-4045-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:48:19.059010 13839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:48:19.060693 13839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:48:19.071483 13844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:48:19.071449 13845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:48:19.071883 13847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:48:19.072304 13839 server_base.cc:1034] running on GCE node
I20260619 18:48:19.072992 13839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:48:19.074230 13839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:48:19.075429 13839 hybrid_clock.cc:648] HybridClock initialized: now 1781894899075382 us; error 41 us; skew 500 ppm
I20260619 18:48:19.075745 13839 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 18:48:19.077786 13839 webserver.cc:460] Webserver started at http://127.3.243.126:35305/ using document root <none> and password file <none>
I20260619 18:48:19.078413 13839 fs_manager.cc:359] Metadata directory not provided
I20260619 18:48:19.078506 13839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:48:19.078794 13839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:48:19.081525 13839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894810715282-4045-0/minicluster-data/master-0/data/instance:
uuid: "cb3b275267b74891a2d193f22c7b4f59"
format_stamp: "Formatted at 2026-06-19 18:48:19 on dist-test-slave-m9wg"
server_key: "a8c158266394f717c742cd7b99fb8252"
server_key_iv: "2a270f63b42c45cf2448287fbdae5a23"
server_key_version: "encryptionkey@0"
I20260619 18:48:19.082252 13839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894810715282-4045-0/minicluster-data/master-0/wal/instance:
uuid: "cb3b275267b74891a2d193f22c7b4f59"
format_stamp: "Formatted at 2026-06-19 18:48:19 on dist-test-slave-m9wg"
server_key: "a8c158266394f717c742cd7b99fb8252"
server_key_iv: "2a270f63b42c45cf2448287fbdae5a23"
server_key_version: "encryptionkey@0"
I20260619 18:48:19.087023 13839 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 18:48:19.090197 13853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:48:19.091441 13839 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 18:48:19.091594 13839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894810715282-4045-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894810715282-4045-0/minicluster-data/master-0/wal
uuid: "cb3b275267b74891a2d193f22c7b4f59"
format_stamp: "Formatted at 2026-06-19 18:48:19 on dist-test-slave-m9wg"
server_key: "a8c158266394f717c742cd7b99fb8252"
server_key_iv: "2a270f63b42c45cf2448287fbdae5a23"
server_key_version: "encryptionkey@0"
I20260619 18:48:19.091779 13839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894810715282-4045-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894810715282-4045-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894810715282-4045-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:48:19.097394 13839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:48:19.098104 13839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:48:19.098361 13839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:48:19.114787 13839 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.126:43507
I20260619 18:48:19.114806 13904 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.126:43507 every 8 connection(s)
I20260619 18:48:19.116132 13839 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
I20260619 18:48:19.119702 13905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:48:19.125959 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 13839
I20260619 18:48:19.126142 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894810715282-4045-0/minicluster-data/master-0/wal/instance
I20260619 18:48:19.126741 4045 external_mini_cluster.cc:1428] Setting key 82eb720c49bedd3ded68e751b3d1a878
I20260619 18:48:19.129722 13905 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb3b275267b74891a2d193f22c7b4f59: Bootstrap starting.
I20260619 18:48:19.132905 13905 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb3b275267b74891a2d193f22c7b4f59: Neither blocks nor log segments found. Creating new log.
I20260619 18:48:19.134102 13905 log.cc:824] T 00000000000000000000000000000000 P cb3b275267b74891a2d193f22c7b4f59: Log is configured to *not* fsync() on all Append() calls
I20260619 18:48:19.134922 13908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:48:19.128793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55948 (local address 127.3.243.126:43507)
0619 18:48:19.129712 (+ 919us) server_negotiation.cc:206] Beginning negotiation
0619 18:48:19.129723 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 18:48:19.129767 (+ 44us) server_negotiation.cc:407] Connection header received
0619 18:48:19.131431 (+ 1664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:48:19.131456 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:48:19.131861 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:48:19.132193 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:48:19.133307 (+ 1114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:48:19.133313 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:48:19.133326 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:48:19.133350 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:48:19.133412 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:48:19.133429 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:48:19.133570 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 18:48:19.133705 (+ 135us) server_negotiation.cc:299] Negotiation successful
0619 18:48:19.133951 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":294,"threads_started":1}
I20260619 18:48:19.136897 13905 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb3b275267b74891a2d193f22c7b4f59: No bootstrap required, opened a new log
I20260619 18:48:19.143956 13905 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb3b275267b74891a2d193f22c7b4f59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb3b275267b74891a2d193f22c7b4f59" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 43507 } }
I20260619 18:48:19.144294 13905 raft_consensus.cc:374] T 00000000000000000000000000000000 P cb3b275267b74891a2d193f22c7b4f59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:48:19.144394 13905 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb3b275267b74891a2d193f22c7b4f59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb3b275267b74891a2d193f22c7b4f59, State: Initialized, Role: FOLLOWER
I20260619 18:48:19.144963 13905 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb3b275267b74891a2d193f22c7b4f59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb3b275267b74891a2d193f22c7b4f59" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 43507 } }
I20260619 18:48:19.145164 13905 raft_consensus.cc:388] T 00000000000000000000000000000000 P cb3b275267b74891a2d193f22c7b4f59 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:48:19.145284 13905 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb3b275267b74891a2d193f22c7b4f59 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:48:19.145434 13905 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb3b275267b74891a2d193f22c7b4f59 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:48:19.147734 13905 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb3b275267b74891a2d193f22c7b4f59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb3b275267b74891a2d193f22c7b4f59" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 43507 } }
I20260619 18:48:19.148245 13905 leader_election.cc:302] T 00000000000000000000000000000000 P cb3b275267b74891a2d193f22c7b4f59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb3b275267b74891a2d193f22c7b4f59; no voters:
I20260619 18:48:19.148767 13905 leader_election.cc:288] T 00000000000000000000000000000000 P cb3b275267b74891a2d193f22c7b4f59 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:48:19.148921 13910 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb3b275267b74891a2d193f22c7b4f59 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:48:19.149574 13910 raft_consensus.cc:686] T 00000000000000000000000000000000 P cb3b275267b74891a2d193f22c7b4f59 [term 1 LEADER]: Becoming Leader. State: Replica: cb3b275267b74891a2d193f22c7b4f59, State: Running, Role: LEADER
I20260619 18:48:19.150067 13910 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb3b275267b74891a2d193f22c7b4f59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb3b275267b74891a2d193f22c7b4f59" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 43507 } }
I20260619 18:48:19.150683 13905 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb3b275267b74891a2d193f22c7b4f59 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:48:19.152592 13912 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb3b275267b74891a2d193f22c7b4f59 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb3b275267b74891a2d193f22c7b4f59. Latest consensus state: current_term: 1 leader_uuid: "cb3b275267b74891a2d193f22c7b4f59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb3b275267b74891a2d193f22c7b4f59" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 43507 } } }
I20260619 18:48:19.152980 13912 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb3b275267b74891a2d193f22c7b4f59 [sys.catalog]: This master's current role is: LEADER
I20260619 18:48:19.153530 13911 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb3b275267b74891a2d193f22c7b4f59 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cb3b275267b74891a2d193f22c7b4f59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb3b275267b74891a2d193f22c7b4f59" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 43507 } } }
I20260619 18:48:19.153925 13911 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb3b275267b74891a2d193f22c7b4f59 [sys.catalog]: This master's current role is: LEADER
I20260619 18:48:19.155360 13918 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:48:19.160358 13918 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:48:19.169095 13918 catalog_manager.cc:1347] Generated new cluster ID: 17e0c813f3b245f8a6195ffca462bb18
I20260619 18:48:19.169194 13918 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:48:19.182775 13918 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:48:19.183157 13918 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:48:19.188540 13918 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cb3b275267b74891a2d193f22c7b4f59: Generated new TSK 0
I20260619 18:48:19.189318 13918 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:48:19.193682 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894810715282-4045-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894810715282-4045-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894810715282-4045-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894810715282-4045-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.65:0
--local_ip_for_outbound_sockets=127.3.243.65
--webserver_interface=127.3.243.65
--webserver_port=0
--tserver_master_addrs=127.3.243.126:43507
--builtin_ntp_servers=127.3.243.84:33861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 18:48:19.527789 13929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:48:19.528066 13929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:48:19.528177 13929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:48:19.535467 13929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:48:19.535584 13929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:48:19.535702 13929 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.65
I20260619 18:48:19.544816 13929 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:33861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894810715282-4045-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894810715282-4045-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894810715282-4045-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.243.65
--webserver_port=0
--tserver_master_addrs=127.3.243.126:43507
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.65
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894810715282-4045-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:48:19.546049 13929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:48:19.547772 13929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:48:19.560554 13934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:48:19.560492 13937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:48:19.560525 13935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:48:19.561853 13929 server_base.cc:1034] running on GCE node
I20260619 18:48:19.562433 13929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:48:19.563481 13929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:48:19.564664 13929 hybrid_clock.cc:648] HybridClock initialized: now 1781894899564613 us; error 45 us; skew 500 ppm
I20260619 18:48:19.564991 13929 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 18:48:19.567181 13929 webserver.cc:460] Webserver started at http://127.3.243.65:37061/ using document root <none> and password file <none>
I20260619 18:48:19.567849 13929 fs_manager.cc:359] Metadata directory not provided
I20260619 18:48:19.567937 13929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:48:19.568285 13929 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:48:19.571167 13929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894810715282-4045-0/minicluster-data/ts-0/data/instance:
uuid: "43939f9e71dd47918ef86b0308cb226f"
format_stamp: "Formatted at 2026-06-19 18:48:19 on dist-test-slave-m9wg"
server_key: "43ef098ccc70d5d44811ddca776f32c6"
server_key_iv: "a7f165ee0ed091c9a77eb7ee1938eaec"
server_key_version: "encryptionkey@0"
I20260619 18:48:19.571923 13929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894810715282-4045-0/minicluster-data/ts-0/wal/instance:
uuid: "43939f9e71dd47918ef86b0308cb226f"
format_stamp: "Formatted at 2026-06-19 18:48:19 on dist-test-slave-m9wg"
server_key: "43ef098ccc70d5d44811ddca776f32c6"
server_key_iv: "a7f165ee0ed091c9a77eb7ee1938eaec"
server_key_version: "encryptionkey@0"
I20260619 18:48:19.577028 13929 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 18:48:19.580463 13943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:48:19.581796 13929 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 18:48:19.581995 13929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894810715282-4045-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894810715282-4045-0/minicluster-data/ts-0/wal
uuid: "43939f9e71dd47918ef86b0308cb226f"
format_stamp: "Formatted at 2026-06-19 18:48:19 on dist-test-slave-m9wg"
server_key: "43ef098ccc70d5d44811ddca776f32c6"
server_key_iv: "a7f165ee0ed091c9a77eb7ee1938eaec"
server_key_version: "encryptionkey@0"
I20260619 18:48:19.582188 13929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894810715282-4045-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894810715282-4045-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894810715282-4045-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:48:19.587685 13929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:48:19.588485 13929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:48:19.588717 13929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:48:19.589576 13929 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:48:19.590965 13929 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:48:19.591059 13929 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:48:19.591164 13929 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:48:19.591231 13929 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:48:19.619920 13929 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.65:37565
I20260619 18:48:19.619971 14055 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.65:37565 every 8 connection(s)
I20260619 18:48:19.621284 13929 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894810715282-4045-0/minicluster-data/ts-0/data/info.pb
I20260619 18:48:19.628849 13908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:48:19.623701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.65:58263 (local address 127.3.243.126:43507)
0619 18:48:19.623871 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 18:48:19.623875 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:48:19.625250 (+ 1375us) server_negotiation.cc:407] Connection header received
0619 18:48:19.626877 (+ 1627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:48:19.626884 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:48:19.626954 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:48:19.627061 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:48:19.628295 (+ 1234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:48:19.628299 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:48:19.628303 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:48:19.628330 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:48:19.628362 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:48:19.628373 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:48:19.628473 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 18:48:19.628596 (+ 123us) server_negotiation.cc:299] Negotiation successful
0619 18:48:19.628645 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 18:48:19.629753 14058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:48:19.624218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:43507 (local address 127.3.243.65:58263)
0619 18:48:19.625096 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0619 18:48:19.625142 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 18:48:19.626613 (+ 1471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:48:19.627234 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:48:19.627242 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:48:19.627355 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:48:19.627995 (+ 640us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:48:19.627997 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:48:19.628035 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:48:19.628117 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:48:19.628445 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:48:19.628451 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:48:19.628456 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 18:48:19.628654 (+ 198us) client_negotiation.cc:240] Negotiation successful
0619 18:48:19.628848 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":238,"threads_started":1}
I20260619 18:48:19.631129 14056 heartbeater.cc:347] Connected to a master server at 127.3.243.126:43507
I20260619 18:48:19.631397 14056 heartbeater.cc:464] Registering TS with master...
I20260619 18:48:19.631577 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 13929
I20260619 18:48:19.631763 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894810715282-4045-0/minicluster-data/ts-0/wal/instance
I20260619 18:48:19.631968 14056 heartbeater.cc:511] Master 127.3.243.126:43507 requested a full tablet report, sending...
I20260619 18:48:19.632457 4045 external_mini_cluster.cc:1428] Setting key 69c523a6e65afffe623bf7e05d4518ec
I20260619 18:48:19.634022 13870 ts_manager.cc:194] Registered new tserver with Master: 43939f9e71dd47918ef86b0308cb226f (127.3.243.65:37565)
I20260619 18:48:19.635560 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894810715282-4045-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894810715282-4045-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894810715282-4045-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894810715282-4045-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.66:0
--local_ip_for_outbound_sockets=127.3.243.66
--webserver_interface=127.3.243.66
--webserver_port=0
--tserver_master_addrs=127.3.243.126:43507
--builtin_ntp_servers=127.3.243.84:33861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 18:48:19.946391 14059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:48:19.946651 14059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:48:19.946776 14059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:48:19.953881 14059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:48:19.953971 14059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:48:19.954110 14059 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.66
I20260619 18:48:19.962798 14059 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:33861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894810715282-4045-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894810715282-4045-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894810715282-4045-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.243.66
--webserver_port=0
--tserver_master_addrs=127.3.243.126:43507
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.66
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894810715282-4045-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:48:19.963997 14059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:48:19.965714 14059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:48:19.977770 14065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:48:19.977886 14067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:48:19.977936 14064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:48:19.978736 14059 server_base.cc:1034] running on GCE node
I20260619 18:48:19.979260 14059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:48:19.980121 14059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:48:19.981349 14059 hybrid_clock.cc:648] HybridClock initialized: now 1781894899981302 us; error 40 us; skew 500 ppm
I20260619 18:48:19.981635 14059 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 18:48:19.983635 14059 webserver.cc:460] Webserver started at http://127.3.243.66:39195/ using document root <none> and password file <none>
I20260619 18:48:19.984326 14059 fs_manager.cc:359] Metadata directory not provided
I20260619 18:48:19.984423 14059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:48:19.984714 14059 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:48:19.987398 14059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894810715282-4045-0/minicluster-data/ts-1/data/instance:
uuid: "97393d1b6e004460a39ab626e4443cd6"
format_stamp: "Formatted at 2026-06-19 18:48:19 on dist-test-slave-m9wg"
server_key: "46b1b2990fe5fea641386bfae25a1a93"
server_key_iv: "2fc506a9d11e707b4812e07420fe4e09"
server_key_version: "encryptionkey@0"
I20260619 18:48:19.988122 14059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894810715282-4045-0/minicluster-data/ts-1/wal/instance:
uuid: "97393d1b6e004460a39ab626e4443cd6"
format_stamp: "Formatted at 2026-06-19 18:48:19 on dist-test-slave-m9wg"
server_key: "46b1b2990fe5fea641386bfae25a1a93"
server_key_iv: "2fc506a9d11e707b4812e07420fe4e09"
server_key_version: "encryptionkey@0"
I20260619 18:48:19.993037 14059 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260619 18:48:19.996395 14073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:48:19.997639 14059 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 18:48:19.997778 14059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894810715282-4045-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894810715282-4045-0/minicluster-data/ts-1/wal
uuid: "97393d1b6e004460a39ab626e4443cd6"
format_stamp: "Formatted at 2026-06-19 18:48:19 on dist-test-slave-m9wg"
server_key: "46b1b2990fe5fea641386bfae25a1a93"
server_key_iv: "2fc506a9d11e707b4812e07420fe4e09"
server_key_version: "encryptionkey@0"
I20260619 18:48:19.997972 14059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894810715282-4045-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894810715282-4045-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894810715282-4045-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:48:20.003078 14059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:48:20.003782 14059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:48:20.004014 14059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:48:20.004907 14059 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:48:20.006191 14059 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:48:20.006280 14059 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:48:20.006382 14059 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:48:20.006448 14059 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:48:20.035113 14059 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.66:43843
I20260619 18:48:20.035156 14185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.66:43843 every 8 connection(s)
I20260619 18:48:20.036470 14059 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894810715282-4045-0/minicluster-data/ts-1/data/info.pb
I20260619 18:48:20.044291 13908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:48:20.038875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.66:58833 (local address 127.3.243.126:43507)
0619 18:48:20.039039 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 18:48:20.039044 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:48:20.040466 (+ 1422us) server_negotiation.cc:407] Connection header received
0619 18:48:20.042076 (+ 1610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:48:20.042082 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:48:20.042151 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:48:20.042239 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:48:20.043504 (+ 1265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:48:20.043511 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:48:20.043516 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:48:20.043534 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:48:20.043567 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:48:20.043578 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:48:20.043691 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 18:48:20.043915 (+ 224us) server_negotiation.cc:299] Negotiation successful
0619 18:48:20.043987 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 18:48:20.044996 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 14059
I20260619 18:48:20.045162 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894810715282-4045-0/minicluster-data/ts-1/wal/instance
I20260619 18:48:20.045150 14188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:48:20.039423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:43507 (local address 127.3.243.66:58833)
0619 18:48:20.040318 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0619 18:48:20.040359 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 18:48:20.041829 (+ 1470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:48:20.042465 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:48:20.042473 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:48:20.042567 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:48:20.043221 (+ 654us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:48:20.043223 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:48:20.043257 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:48:20.043341 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:48:20.043659 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:48:20.043670 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:48:20.043677 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 18:48:20.043982 (+ 305us) client_negotiation.cc:240] Negotiation successful
0619 18:48:20.044253 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":242,"threads_started":1}
I20260619 18:48:20.045715 4045 external_mini_cluster.cc:1428] Setting key 6c9b98b325cfd48c6b1241d0c87030b9
I20260619 18:48:20.046471 14186 heartbeater.cc:347] Connected to a master server at 127.3.243.126:43507
I20260619 18:48:20.046751 14186 heartbeater.cc:464] Registering TS with master...
I20260619 18:48:20.047309 14186 heartbeater.cc:511] Master 127.3.243.126:43507 requested a full tablet report, sending...
I20260619 18:48:20.048465 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894810715282-4045-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894810715282-4045-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894810715282-4045-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894810715282-4045-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.67:0
--local_ip_for_outbound_sockets=127.3.243.67
--webserver_interface=127.3.243.67
--webserver_port=0
--tserver_master_addrs=127.3.243.126:43507
--builtin_ntp_servers=127.3.243.84:33861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260619 18:48:20.048745 13870 ts_manager.cc:194] Registered new tserver with Master: 97393d1b6e004460a39ab626e4443cd6 (127.3.243.66:43843)
W20260619 18:48:20.358392 14189 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:48:20.358647 14189 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:48:20.358757 14189 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:48:20.365994 14189 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:48:20.366107 14189 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:48:20.366273 14189 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.67
I20260619 18:48:20.375849 14189 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:33861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894810715282-4045-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894810715282-4045-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894810715282-4045-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.243.67
--webserver_port=0
--tserver_master_addrs=127.3.243.126:43507
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.67
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894810715282-4045-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:48:20.377079 14189 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:48:20.378743 14189 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:48:20.390936 14194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:48:20.390890 14197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:48:20.390964 14195 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:48:20.391659 14189 server_base.cc:1034] running on GCE node
I20260619 18:48:20.392236 14189 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:48:20.393078 14189 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:48:20.394277 14189 hybrid_clock.cc:648] HybridClock initialized: now 1781894900394220 us; error 48 us; skew 500 ppm
I20260619 18:48:20.394573 14189 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 18:48:20.396792 14189 webserver.cc:460] Webserver started at http://127.3.243.67:41091/ using document root <none> and password file <none>
I20260619 18:48:20.397459 14189 fs_manager.cc:359] Metadata directory not provided
I20260619 18:48:20.397555 14189 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:48:20.397862 14189 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:48:20.400527 14189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894810715282-4045-0/minicluster-data/ts-2/data/instance:
uuid: "17b24cad147b40b58068d3c8787a5800"
format_stamp: "Formatted at 2026-06-19 18:48:20 on dist-test-slave-m9wg"
server_key: "efcea56d41b10ca6bf0633bab90979fe"
server_key_iv: "d228d087966f1cffda41e3c450e053a2"
server_key_version: "encryptionkey@0"
I20260619 18:48:20.401259 14189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894810715282-4045-0/minicluster-data/ts-2/wal/instance:
uuid: "17b24cad147b40b58068d3c8787a5800"
format_stamp: "Formatted at 2026-06-19 18:48:20 on dist-test-slave-m9wg"
server_key: "efcea56d41b10ca6bf0633bab90979fe"
server_key_iv: "d228d087966f1cffda41e3c450e053a2"
server_key_version: "encryptionkey@0"
I20260619 18:48:20.406111 14189 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 18:48:20.409350 14203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:48:20.410557 14189 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 18:48:20.410708 14189 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894810715282-4045-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894810715282-4045-0/minicluster-data/ts-2/wal
uuid: "17b24cad147b40b58068d3c8787a5800"
format_stamp: "Formatted at 2026-06-19 18:48:20 on dist-test-slave-m9wg"
server_key: "efcea56d41b10ca6bf0633bab90979fe"
server_key_iv: "d228d087966f1cffda41e3c450e053a2"
server_key_version: "encryptionkey@0"
I20260619 18:48:20.410900 14189 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894810715282-4045-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894810715282-4045-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894810715282-4045-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:48:20.416016 14189 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:48:20.416775 14189 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:48:20.417004 14189 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:48:20.417840 14189 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:48:20.419085 14189 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:48:20.419174 14189 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:48:20.419283 14189 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:48:20.419353 14189 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:48:20.446497 14189 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.67:35805
I20260619 18:48:20.446537 14315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.67:35805 every 8 connection(s)
I20260619 18:48:20.447710 14189 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894810715282-4045-0/minicluster-data/ts-2/data/info.pb
I20260619 18:48:20.448510 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 14189
I20260619 18:48:20.448685 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894810715282-4045-0/minicluster-data/ts-2/wal/instance
I20260619 18:48:20.449160 4045 external_mini_cluster.cc:1428] Setting key c5e48f476b9b268c952c1990932353d4
I20260619 18:48:20.455446 13908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:48:20.450214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.67:56331 (local address 127.3.243.126:43507)
0619 18:48:20.450403 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 18:48:20.450410 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:48:20.452011 (+ 1601us) server_negotiation.cc:407] Connection header received
0619 18:48:20.453633 (+ 1622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:48:20.453637 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:48:20.453686 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:48:20.453760 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:48:20.454937 (+ 1177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:48:20.454940 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:48:20.454943 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:48:20.454955 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:48:20.454980 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:48:20.454989 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:48:20.455067 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 18:48:20.455197 (+ 130us) server_negotiation.cc:299] Negotiation successful
0619 18:48:20.455242 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 18:48:20.456406 14318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:48:20.450824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:43507 (local address 127.3.243.67:56331)
0619 18:48:20.451860 (+ 1036us) negotiation.cc:107] Waiting for socket to connect
0619 18:48:20.451910 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 18:48:20.453412 (+ 1502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:48:20.453971 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:48:20.453979 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:48:20.454068 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:48:20.454687 (+ 619us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:48:20.454690 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:48:20.454734 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:48:20.454817 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:48:20.455055 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:48:20.455061 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:48:20.455066 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 18:48:20.455272 (+ 206us) client_negotiation.cc:240] Negotiation successful
0619 18:48:20.455457 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":292,"threads_started":1}
I20260619 18:48:20.457638 14316 heartbeater.cc:347] Connected to a master server at 127.3.243.126:43507
I20260619 18:48:20.457904 14316 heartbeater.cc:464] Registering TS with master...
I20260619 18:48:20.458441 14316 heartbeater.cc:511] Master 127.3.243.126:43507 requested a full tablet report, sending...
I20260619 18:48:20.459664 13870 ts_manager.cc:194] Registered new tserver with Master: 17b24cad147b40b58068d3c8787a5800 (127.3.243.67:35805)
I20260619 18:48:20.463977 4045 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 18:48:20.469506 13908 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:48:20.467978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41424 (local address 127.3.243.126:43507)
0619 18:48:20.468125 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 18:48:20.468130 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:48:20.468617 (+ 487us) server_negotiation.cc:407] Connection header received
0619 18:48:20.468781 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:48:20.468787 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:48:20.468840 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:48:20.468912 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:48:20.469261 (+ 349us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41424: BlockingRecv error: recv got EOF from 127.0.0.1:41424 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 18:48:20.469956 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 13929
I20260619 18:48:20.473242 14325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:48:20.470919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59352 (local address 127.3.243.65:37565)
0619 18:48:20.471386 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0619 18:48:20.471393 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:48:20.471416 (+ 23us) server_negotiation.cc:407] Connection header received
0619 18:48:20.471468 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:48:20.471474 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:48:20.471649 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:48:20.471762 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:48:20.472641 (+ 879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:48:20.472647 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:48:20.472658 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:48:20.472674 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:48:20.472713 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:48:20.472726 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:48:20.472837 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 18:48:20.472891 (+ 54us) server_negotiation.cc:299] Negotiation successful
0619 18:48:20.472971 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":210,"threads_started":1}
I20260619 18:48:20.474346 14051 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:48:20.553913 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 13929
I20260619 18:48:20.567236 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 14059
I20260619 18:48:20.570626 14327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:48:20.568106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50716 (local address 127.3.243.66:43843)
0619 18:48:20.568607 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0619 18:48:20.568614 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:48:20.568642 (+ 28us) server_negotiation.cc:407] Connection header received
0619 18:48:20.568705 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:48:20.568711 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:48:20.568933 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:48:20.569061 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:48:20.569935 (+ 874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:48:20.569942 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:48:20.569954 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:48:20.569970 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:48:20.570009 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:48:20.570023 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:48:20.570158 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 18:48:20.570249 (+ 91us) server_negotiation.cc:299] Negotiation successful
0619 18:48:20.570346 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":223,"threads_started":1}
I20260619 18:48:20.571875 14181 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:48:20.653501 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 14059
I20260619 18:48:20.666280 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 14189
I20260619 18:48:20.670056 14329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:48:20.667209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50364 (local address 127.3.243.67:35805)
0619 18:48:20.667776 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0619 18:48:20.667783 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:48:20.667813 (+ 30us) server_negotiation.cc:407] Connection header received
0619 18:48:20.667895 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:48:20.667901 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:48:20.668109 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:48:20.668286 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:48:20.669349 (+ 1063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:48:20.669357 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:48:20.669370 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:48:20.669382 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:48:20.669424 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:48:20.669442 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:48:20.669594 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 18:48:20.669666 (+ 72us) server_negotiation.cc:299] Negotiation successful
0619 18:48:20.669771 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":254,"threads_started":1}
I20260619 18:48:20.671241 14311 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:48:20.750049 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 14189
I20260619 18:48:20.762274 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 13839
I20260619 18:48:20.763437 13900 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:48:20.864121 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 13839
2026-06-19T18:48:20Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2189 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894810715282-4045-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:48:20 dist-test-slave-m9wg krb5kdc[14334](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:48:20 dist-test-slave-m9wg krb5kdc[14334](info): set up 2 sockets
Jun 19 18:48:20 dist-test-slave-m9wg krb5kdc[14334](info): commencing operation
krb5kdc: starting...
W20260619 18:48:21.872854 4045 mini_kdc.cc:121] Time spent starting KDC: real 0.973s user 0.000s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894810715282-4045-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894810715282-4045-0/krb5kdc/test-user.keytab.
Jun 19 18:48:22 dist-test-slave-m9wg krb5kdc[14334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894902, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:48:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:48:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.243.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab.
Entry for principal kudu/127.3.243.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab.
I20260619 18:48:22.068734 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894810715282-4045-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894810715282-4045-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894810715282-4045-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.126:37933
--webserver_interface=127.3.243.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab
--principal=kudu/127.3.243.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:32807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.126:37933
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:48:22.380115 14347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:48:22.380460 14347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:48:22.380589 14347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:48:22.387687 14347 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:48:22.387813 14347 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:48:22.387889 14347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:48:22.387962 14347 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:48:22.388018 14347 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:48:22.397758 14347 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:32807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894810715282-4045-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894810715282-4045-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.243.126:37933
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab
--principal=kudu/127.3.243.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.126:37933
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.243.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894810715282-4045-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:48:22.399035 14347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:48:22.400350 14347 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:48:22.411369 14352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:48:22.411324 14353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:48:22.411405 14355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:48:22.412410 14347 server_base.cc:1034] running on GCE node
I20260619 18:48:22.413117 14347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:48:22.414454 14347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:48:22.415656 14347 hybrid_clock.cc:648] HybridClock initialized: now 1781894902415612 us; error 36 us; skew 500 ppm
I20260619 18:48:22.415997 14347 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:48:22 dist-test-slave-m9wg krb5kdc[14334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894902, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:48:22.418929 14347 init.cc:377] Logged in from keytab as kudu/127.3.243.126@KRBTEST.COM (short username kudu)
I20260619 18:48:22.420279 14347 webserver.cc:460] Webserver started at http://127.3.243.126:40339/ using document root <none> and password file <none>
I20260619 18:48:22.420905 14347 fs_manager.cc:359] Metadata directory not provided
I20260619 18:48:22.421018 14347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:48:22.421300 14347 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:48:22.424073 14347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894810715282-4045-0/minicluster-data/master-0/data/instance:
uuid: "7533f699512645be834b42093f86d2e4"
format_stamp: "Formatted at 2026-06-19 18:48:22 on dist-test-slave-m9wg"
server_key: "b59f800fb5f4383919e074cbc827bd55"
server_key_iv: "7ebbfa1a69a043066af0b4f0e6b8f422"
server_key_version: "encryptionkey@0"
I20260619 18:48:22.424840 14347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894810715282-4045-0/minicluster-data/master-0/wal/instance:
uuid: "7533f699512645be834b42093f86d2e4"
format_stamp: "Formatted at 2026-06-19 18:48:22 on dist-test-slave-m9wg"
server_key: "b59f800fb5f4383919e074cbc827bd55"
server_key_iv: "7ebbfa1a69a043066af0b4f0e6b8f422"
server_key_version: "encryptionkey@0"
I20260619 18:48:22.429659 14347 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 18:48:22.432915 14362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:48:22.434188 14347 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 18:48:22.434345 14347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894810715282-4045-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894810715282-4045-0/minicluster-data/master-0/wal
uuid: "7533f699512645be834b42093f86d2e4"
format_stamp: "Formatted at 2026-06-19 18:48:22 on dist-test-slave-m9wg"
server_key: "b59f800fb5f4383919e074cbc827bd55"
server_key_iv: "7ebbfa1a69a043066af0b4f0e6b8f422"
server_key_version: "encryptionkey@0"
I20260619 18:48:22.434510 14347 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894810715282-4045-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894810715282-4045-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894810715282-4045-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:48:22.450358 14347 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:48:22.454310 14347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:48:22.454557 14347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:48:22.470860 14347 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.126:37933
I20260619 18:48:22.470877 14413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.126:37933 every 8 connection(s)
I20260619 18:48:22.472179 14347 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
I20260619 18:48:22.475639 14414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:48:22.478067 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 14347
I20260619 18:48:22.478283 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894810715282-4045-0/minicluster-data/master-0/wal/instance
I20260619 18:48:22.479036 4045 external_mini_cluster.cc:1428] Setting key 9fb5aa259fde121333ca5ee1e20d977f
I20260619 18:48:22.487818 14414 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7533f699512645be834b42093f86d2e4: Bootstrap starting.
I20260619 18:48:22.491441 14414 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7533f699512645be834b42093f86d2e4: Neither blocks nor log segments found. Creating new log.
I20260619 18:48:22.492729 14414 log.cc:824] T 00000000000000000000000000000000 P 7533f699512645be834b42093f86d2e4: Log is configured to *not* fsync() on all Append() calls
Jun 19 18:48:22 dist-test-slave-m9wg krb5kdc[14334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894902, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.126@KRBTEST.COM
I20260619 18:48:22.495494 14414 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7533f699512645be834b42093f86d2e4: No bootstrap required, opened a new log
I20260619 18:48:22.501130 14417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:48:22.481054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45286 (local address 127.3.243.126:37933)
0619 18:48:22.481760 (+ 706us) server_negotiation.cc:206] Beginning negotiation
0619 18:48:22.481771 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 18:48:22.481810 (+ 39us) server_negotiation.cc:407] Connection header received
0619 18:48:22.483406 (+ 1596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:48:22.483429 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:48:22.483807 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:48:22.484170 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:48:22.486086 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:22.487310 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:22.488967 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:22.489618 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:22.489745 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:22.494269 (+ 4524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:48:22.494277 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:48:22.494291 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:48:22.494324 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:48:22.497223 (+ 2899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:48:22.497864 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:48:22.497870 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:48:22.497879 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:48:22.498045 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:48:22.498398 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:48:22.498401 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:48:22.498403 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:48:22.498948 (+ 545us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:48:22.499255 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0619 18:48:22.499642 (+ 387us) server_negotiation.cc:299] Negotiation successful
0619 18:48:22.500044 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":330,"threads_started":1}
I20260619 18:48:22.503146 14414 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7533f699512645be834b42093f86d2e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7533f699512645be834b42093f86d2e4" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 37933 } }
I20260619 18:48:22.503490 14414 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7533f699512645be834b42093f86d2e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:48:22.503590 14414 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7533f699512645be834b42093f86d2e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7533f699512645be834b42093f86d2e4, State: Initialized, Role: FOLLOWER
I20260619 18:48:22.504132 14414 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7533f699512645be834b42093f86d2e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7533f699512645be834b42093f86d2e4" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 37933 } }
I20260619 18:48:22.504370 14414 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7533f699512645be834b42093f86d2e4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:48:22.504488 14414 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7533f699512645be834b42093f86d2e4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:48:22.504623 14414 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7533f699512645be834b42093f86d2e4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:48:22.506662 14414 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7533f699512645be834b42093f86d2e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7533f699512645be834b42093f86d2e4" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 37933 } }
I20260619 18:48:22.507122 14414 leader_election.cc:302] T 00000000000000000000000000000000 P 7533f699512645be834b42093f86d2e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7533f699512645be834b42093f86d2e4; no voters:
I20260619 18:48:22.507663 14414 leader_election.cc:288] T 00000000000000000000000000000000 P 7533f699512645be834b42093f86d2e4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:48:22.507767 14419 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7533f699512645be834b42093f86d2e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:48:22.508798 14419 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7533f699512645be834b42093f86d2e4 [term 1 LEADER]: Becoming Leader. State: Replica: 7533f699512645be834b42093f86d2e4, State: Running, Role: LEADER
I20260619 18:48:22.509338 14419 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7533f699512645be834b42093f86d2e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7533f699512645be834b42093f86d2e4" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 37933 } }
I20260619 18:48:22.509771 14414 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7533f699512645be834b42093f86d2e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:48:22.512070 14420 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7533f699512645be834b42093f86d2e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7533f699512645be834b42093f86d2e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7533f699512645be834b42093f86d2e4" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 37933 } } }
I20260619 18:48:22.512125 14421 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7533f699512645be834b42093f86d2e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7533f699512645be834b42093f86d2e4. Latest consensus state: current_term: 1 leader_uuid: "7533f699512645be834b42093f86d2e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7533f699512645be834b42093f86d2e4" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 37933 } } }
I20260619 18:48:22.512446 14420 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7533f699512645be834b42093f86d2e4 [sys.catalog]: This master's current role is: LEADER
I20260619 18:48:22.512508 14421 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7533f699512645be834b42093f86d2e4 [sys.catalog]: This master's current role is: LEADER
I20260619 18:48:22.513427 14427 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:48:22.519228 14427 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:48:22.528992 14427 catalog_manager.cc:1347] Generated new cluster ID: b2cd0c5f660c41f1851a6b6ff2cd4a79
I20260619 18:48:22.529098 14427 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:48:22.551316 14427 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:48:22.553210 14427 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:48:22.573606 14427 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7533f699512645be834b42093f86d2e4: Generated new TSK 0
I20260619 18:48:22.574643 14427 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.243.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.65.keytab.
Entry for principal kudu/127.3.243.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.65.keytab.
I20260619 18:48:22.631382 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894810715282-4045-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894810715282-4045-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894810715282-4045-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894810715282-4045-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.65:0
--local_ip_for_outbound_sockets=127.3.243.65
--webserver_interface=127.3.243.65
--webserver_port=0
--tserver_master_addrs=127.3.243.126:37933
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.65.keytab
--principal=kudu/127.3.243.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:32807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:48:22.978916 14440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:48:22.979200 14440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:48:22.979301 14440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:48:22.986735 14440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:48:22.986868 14440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:48:22.986992 14440 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.65
I20260619 18:48:22.996744 14440 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:32807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894810715282-4045-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894810715282-4045-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.65.keytab
--principal=kudu/127.3.243.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894810715282-4045-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.243.65
--webserver_port=0
--tserver_master_addrs=127.3.243.126:37933
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.65
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894810715282-4045-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:48:22.998040 14440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:48:22.999388 14440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:48:23.013008 14445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:48:23.013034 14446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:48:23.013063 14448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:48:23.013887 14440 server_base.cc:1034] running on GCE node
I20260619 18:48:23.014448 14440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:48:23.015343 14440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:48:23.016535 14440 hybrid_clock.cc:648] HybridClock initialized: now 1781894903016485 us; error 41 us; skew 500 ppm
I20260619 18:48:23.016829 14440 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:48:23 dist-test-slave-m9wg krb5kdc[14334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894903, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:48:23.019917 14440 init.cc:377] Logged in from keytab as kudu/127.3.243.65@KRBTEST.COM (short username kudu)
I20260619 18:48:23.021402 14440 webserver.cc:460] Webserver started at http://127.3.243.65:40779/ using document root <none> and password file <none>
I20260619 18:48:23.022058 14440 fs_manager.cc:359] Metadata directory not provided
I20260619 18:48:23.022171 14440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:48:23.022451 14440 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:48:23.025404 14440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894810715282-4045-0/minicluster-data/ts-0/data/instance:
uuid: "dae9c739628c495bb99a074e73e5bffe"
format_stamp: "Formatted at 2026-06-19 18:48:23 on dist-test-slave-m9wg"
server_key: "7ef4dce7d54958c401c3285bb5a499f9"
server_key_iv: "afcd9a870ce963a20e5fce48745abf0a"
server_key_version: "encryptionkey@0"
I20260619 18:48:23.026151 14440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894810715282-4045-0/minicluster-data/ts-0/wal/instance:
uuid: "dae9c739628c495bb99a074e73e5bffe"
format_stamp: "Formatted at 2026-06-19 18:48:23 on dist-test-slave-m9wg"
server_key: "7ef4dce7d54958c401c3285bb5a499f9"
server_key_iv: "afcd9a870ce963a20e5fce48745abf0a"
server_key_version: "encryptionkey@0"
I20260619 18:48:23.031096 14440 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 18:48:23.034451 14455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:48:23.035732 14440 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 18:48:23.035888 14440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894810715282-4045-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894810715282-4045-0/minicluster-data/ts-0/wal
uuid: "dae9c739628c495bb99a074e73e5bffe"
format_stamp: "Formatted at 2026-06-19 18:48:23 on dist-test-slave-m9wg"
server_key: "7ef4dce7d54958c401c3285bb5a499f9"
server_key_iv: "afcd9a870ce963a20e5fce48745abf0a"
server_key_version: "encryptionkey@0"
I20260619 18:48:23.036062 14440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894810715282-4045-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894810715282-4045-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894810715282-4045-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:48:23.065992 14440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:48:23.069931 14440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:48:23.070199 14440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:48:23.071138 14440 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:48:23.072477 14440 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:48:23.072556 14440 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:48:23.072665 14440 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:48:23.072726 14440 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:48:23.101755 14440 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.65:39555
I20260619 18:48:23.101822 14567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.65:39555 every 8 connection(s)
I20260619 18:48:23.103006 14440 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894810715282-4045-0/minicluster-data/ts-0/data/info.pb
I20260619 18:48:23.109030 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 14440
I20260619 18:48:23.109218 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894810715282-4045-0/minicluster-data/ts-0/wal/instance
I20260619 18:48:23.110008 4045 external_mini_cluster.cc:1428] Setting key 54def6cdff6372ee2be902719f8eb3d3
Jun 19 18:48:23 dist-test-slave-m9wg krb5kdc[14334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894903, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.65@KRBTEST.COM for kudu/127.3.243.126@KRBTEST.COM
I20260619 18:48:23.123107 14570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:48:23.105406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.65:36173 (local address 127.3.243.126:37933)
0619 18:48:23.105820 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0619 18:48:23.105825 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:48:23.106616 (+ 791us) server_negotiation.cc:407] Connection header received
0619 18:48:23.108400 (+ 1784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:48:23.108405 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:48:23.108489 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:48:23.108583 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:48:23.110339 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:23.111279 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:23.113144 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:23.113752 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:23.113888 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:23.117373 (+ 3485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:48:23.117379 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:48:23.117383 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:48:23.117419 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:48:23.120100 (+ 2681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:48:23.120946 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:48:23.120950 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:48:23.120952 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:48:23.121064 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:48:23.121481 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:48:23.121484 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:48:23.121486 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:48:23.121730 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:48:23.121844 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 18:48:23.122680 (+ 836us) server_negotiation.cc:299] Negotiation successful
0619 18:48:23.122858 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":214,"threads_started":1}
I20260619 18:48:23.124392 14571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:48:23.105825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:37933 (local address 127.3.243.65:36173)
0619 18:48:23.106466 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0619 18:48:23.106509 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 18:48:23.108101 (+ 1592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:48:23.108831 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:48:23.108841 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:48:23.109402 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:48:23.110033 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:23.110054 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:23.111452 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:23.111457 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:23.112991 (+ 1534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:23.113003 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:23.113940 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:23.113945 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:23.114138 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:23.114773 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:48:23.114795 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:48:23.117120 (+ 2325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:48:23.120259 (+ 3139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:48:23.120267 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:48:23.120286 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:48:23.120825 (+ 539us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:48:23.121150 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:48:23.121153 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:48:23.121158 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:48:23.121387 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:48:23.121845 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:48:23.121852 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:48:23.122252 (+ 400us) client_negotiation.cc:769] Sending connection context
0619 18:48:23.122543 (+ 291us) client_negotiation.cc:240] Negotiation successful
0619 18:48:23.123055 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":251,"threads_started":1}
I20260619 18:48:23.125909 14568 heartbeater.cc:347] Connected to a master server at 127.3.243.126:37933
I20260619 18:48:23.126200 14568 heartbeater.cc:464] Registering TS with master...
I20260619 18:48:23.126895 14568 heartbeater.cc:511] Master 127.3.243.126:37933 requested a full tablet report, sending...
I20260619 18:48:23.128961 14379 ts_manager.cc:194] Registered new tserver with Master: dae9c739628c495bb99a074e73e5bffe (127.3.243.65:39555)
I20260619 18:48:23.130362 14379 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.65@KRBTEST.COM'} at 127.3.243.65:36173
WARNING: no policy specified for kudu/127.3.243.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.66.keytab.
Entry for principal kudu/127.3.243.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.66.keytab.
I20260619 18:48:23.158982 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894810715282-4045-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894810715282-4045-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894810715282-4045-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894810715282-4045-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.66:0
--local_ip_for_outbound_sockets=127.3.243.66
--webserver_interface=127.3.243.66
--webserver_port=0
--tserver_master_addrs=127.3.243.126:37933
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.66.keytab
--principal=kudu/127.3.243.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:32807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:48:23.465709 14574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:48:23.465999 14574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:48:23.466132 14574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:48:23.474884 14574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:48:23.475011 14574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:48:23.475191 14574 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.66
I20260619 18:48:23.490396 14574 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:32807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894810715282-4045-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894810715282-4045-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.66.keytab
--principal=kudu/127.3.243.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894810715282-4045-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.243.66
--webserver_port=0
--tserver_master_addrs=127.3.243.126:37933
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.66
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894810715282-4045-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:48:23.491961 14574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:48:23.493445 14574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:48:23.505798 14580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:48:23.505848 14579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:48:23.505877 14582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:48:23.506413 14574 server_base.cc:1034] running on GCE node
I20260619 18:48:23.507014 14574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:48:23.507934 14574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:48:23.509157 14574 hybrid_clock.cc:648] HybridClock initialized: now 1781894903509086 us; error 58 us; skew 500 ppm
I20260619 18:48:23.509460 14574 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:48:23 dist-test-slave-m9wg krb5kdc[14334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894903, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:48:23.512647 14574 init.cc:377] Logged in from keytab as kudu/127.3.243.66@KRBTEST.COM (short username kudu)
I20260619 18:48:23.513960 14574 webserver.cc:460] Webserver started at http://127.3.243.66:44769/ using document root <none> and password file <none>
I20260619 18:48:23.514597 14574 fs_manager.cc:359] Metadata directory not provided
I20260619 18:48:23.514709 14574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:48:23.514981 14574 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:48:23.517921 14574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894810715282-4045-0/minicluster-data/ts-1/data/instance:
uuid: "7794eff267c147beaa8a5ba7458bd4ed"
format_stamp: "Formatted at 2026-06-19 18:48:23 on dist-test-slave-m9wg"
server_key: "17ad7af1f783e09ab47bc180c519ac4f"
server_key_iv: "a224e47f1ed3c95a72a172595686fbeb"
server_key_version: "encryptionkey@0"
I20260619 18:48:23.518644 14574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894810715282-4045-0/minicluster-data/ts-1/wal/instance:
uuid: "7794eff267c147beaa8a5ba7458bd4ed"
format_stamp: "Formatted at 2026-06-19 18:48:23 on dist-test-slave-m9wg"
server_key: "17ad7af1f783e09ab47bc180c519ac4f"
server_key_iv: "a224e47f1ed3c95a72a172595686fbeb"
server_key_version: "encryptionkey@0"
I20260619 18:48:23.523422 14574 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 18:48:23.526670 14589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:48:23.527894 14574 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 18:48:23.528016 14574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894810715282-4045-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894810715282-4045-0/minicluster-data/ts-1/wal
uuid: "7794eff267c147beaa8a5ba7458bd4ed"
format_stamp: "Formatted at 2026-06-19 18:48:23 on dist-test-slave-m9wg"
server_key: "17ad7af1f783e09ab47bc180c519ac4f"
server_key_iv: "a224e47f1ed3c95a72a172595686fbeb"
server_key_version: "encryptionkey@0"
I20260619 18:48:23.528198 14574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894810715282-4045-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894810715282-4045-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894810715282-4045-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:48:23.549366 14574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:48:23.553459 14574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:48:23.553704 14574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:48:23.554589 14574 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:48:23.555836 14574 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:48:23.555907 14574 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:48:23.556015 14574 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:48:23.556062 14574 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:48:23.584081 14574 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.66:34971
I20260619 18:48:23.584126 14701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.66:34971 every 8 connection(s)
I20260619 18:48:23.585332 14574 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894810715282-4045-0/minicluster-data/ts-1/data/info.pb
I20260619 18:48:23.587896 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 14574
I20260619 18:48:23.588088 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894810715282-4045-0/minicluster-data/ts-1/wal/instance
I20260619 18:48:23.588685 4045 external_mini_cluster.cc:1428] Setting key 3d8750dbdda9cab09e51ebaaef338665
Jun 19 18:48:23 dist-test-slave-m9wg krb5kdc[14334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894903, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.66@KRBTEST.COM for kudu/127.3.243.126@KRBTEST.COM
I20260619 18:48:23.603885 14570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:48:23.587656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.66:51953 (local address 127.3.243.126:37933)
0619 18:48:23.587821 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 18:48:23.587826 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:48:23.589162 (+ 1336us) server_negotiation.cc:407] Connection header received
0619 18:48:23.591042 (+ 1880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:48:23.591048 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:48:23.591128 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:48:23.591207 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:48:23.592656 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:23.593350 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:23.594805 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:23.595211 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:23.595317 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:23.598881 (+ 3564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:48:23.598886 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:48:23.598889 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:48:23.598920 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:48:23.601198 (+ 2278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:48:23.601841 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:48:23.601844 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:48:23.601846 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:48:23.601966 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:48:23.602344 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:48:23.602347 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:48:23.602349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:48:23.602589 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:48:23.602715 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 18:48:23.603346 (+ 631us) server_negotiation.cc:299] Negotiation successful
0619 18:48:23.603562 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 18:48:23.604794 14704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:48:23.588259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:37933 (local address 127.3.243.66:51953)
0619 18:48:23.589001 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0619 18:48:23.589043 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 18:48:23.590805 (+ 1762us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:48:23.591443 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:48:23.591451 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:48:23.591920 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:48:23.592501 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:23.592523 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:23.593484 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:23.593487 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:23.594669 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:23.594680 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:23.595415 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:23.595422 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:23.595629 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:23.596264 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:48:23.596295 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:48:23.598683 (+ 2388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:48:23.601317 (+ 2634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:48:23.601326 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:48:23.601340 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:48:23.601727 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:48:23.602061 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:48:23.602066 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:48:23.602070 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:48:23.602236 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:48:23.602701 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:48:23.602708 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:48:23.602998 (+ 290us) client_negotiation.cc:769] Sending connection context
0619 18:48:23.603218 (+ 220us) client_negotiation.cc:240] Negotiation successful
0619 18:48:23.603681 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":286,"threads_started":1}
I20260619 18:48:23.606190 14702 heartbeater.cc:347] Connected to a master server at 127.3.243.126:37933
I20260619 18:48:23.606458 14702 heartbeater.cc:464] Registering TS with master...
I20260619 18:48:23.607096 14702 heartbeater.cc:511] Master 127.3.243.126:37933 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.3.243.67@KRBTEST.COM; defaulting to no policy
I20260619 18:48:23.608830 14379 ts_manager.cc:194] Registered new tserver with Master: 7794eff267c147beaa8a5ba7458bd4ed (127.3.243.66:34971)
I20260619 18:48:23.609759 14379 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.66@KRBTEST.COM'} at 127.3.243.66:51953
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.67.keytab.
Entry for principal kudu/127.3.243.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.67.keytab.
I20260619 18:48:23.635099 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894810715282-4045-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894810715282-4045-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894810715282-4045-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894810715282-4045-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.67:0
--local_ip_for_outbound_sockets=127.3.243.67
--webserver_interface=127.3.243.67
--webserver_port=0
--tserver_master_addrs=127.3.243.126:37933
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.67.keytab
--principal=kudu/127.3.243.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:32807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:48:23.938271 14707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:48:23.938525 14707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:48:23.938616 14707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:48:23.945802 14707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:48:23.945904 14707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:48:23.946017 14707 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.67
I20260619 18:48:23.955282 14707 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:32807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894810715282-4045-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894810715282-4045-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.67.keytab
--principal=kudu/127.3.243.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894810715282-4045-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.243.67
--webserver_port=0
--tserver_master_addrs=127.3.243.126:37933
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.67
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894810715282-4045-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:48:23.956487 14707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:48:23.957690 14707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:48:23.970063 14712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:48:23.970103 14713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:48:23.970134 14715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:48:23.970381 14707 server_base.cc:1034] running on GCE node
I20260619 18:48:23.971140 14707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:48:23.971971 14707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:48:23.973169 14707 hybrid_clock.cc:648] HybridClock initialized: now 1781894903973106 us; error 50 us; skew 500 ppm
I20260619 18:48:23.973472 14707 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:48:23 dist-test-slave-m9wg krb5kdc[14334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894903, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:48:23.976334 14707 init.cc:377] Logged in from keytab as kudu/127.3.243.67@KRBTEST.COM (short username kudu)
I20260619 18:48:23.977593 14707 webserver.cc:460] Webserver started at http://127.3.243.67:38105/ using document root <none> and password file <none>
I20260619 18:48:23.978240 14707 fs_manager.cc:359] Metadata directory not provided
I20260619 18:48:23.978353 14707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:48:23.978624 14707 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:48:23.981499 14707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894810715282-4045-0/minicluster-data/ts-2/data/instance:
uuid: "877dcbffd6334d24afdd996375f7becc"
format_stamp: "Formatted at 2026-06-19 18:48:23 on dist-test-slave-m9wg"
server_key: "2f48189860b2c2785949245765572341"
server_key_iv: "3d144a6fa814c9033256ddef355ded72"
server_key_version: "encryptionkey@0"
I20260619 18:48:23.982263 14707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894810715282-4045-0/minicluster-data/ts-2/wal/instance:
uuid: "877dcbffd6334d24afdd996375f7becc"
format_stamp: "Formatted at 2026-06-19 18:48:23 on dist-test-slave-m9wg"
server_key: "2f48189860b2c2785949245765572341"
server_key_iv: "3d144a6fa814c9033256ddef355ded72"
server_key_version: "encryptionkey@0"
I20260619 18:48:23.987134 14707 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 18:48:23.990432 14722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:48:23.991654 14707 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 18:48:23.991803 14707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894810715282-4045-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894810715282-4045-0/minicluster-data/ts-2/wal
uuid: "877dcbffd6334d24afdd996375f7becc"
format_stamp: "Formatted at 2026-06-19 18:48:23 on dist-test-slave-m9wg"
server_key: "2f48189860b2c2785949245765572341"
server_key_iv: "3d144a6fa814c9033256ddef355ded72"
server_key_version: "encryptionkey@0"
I20260619 18:48:23.991972 14707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894810715282-4045-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894810715282-4045-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894810715282-4045-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:48:24.014290 14707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:48:24.018554 14707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:48:24.018815 14707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:48:24.019784 14707 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:48:24.021317 14707 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:48:24.021401 14707 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:48:24.021541 14707 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:48:24.021613 14707 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:48:24.051333 14707 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.67:40497
I20260619 18:48:24.051384 14834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.67:40497 every 8 connection(s)
I20260619 18:48:24.052620 14707 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894810715282-4045-0/minicluster-data/ts-2/data/info.pb
I20260619 18:48:24.053877 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 14707
I20260619 18:48:24.054064 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894810715282-4045-0/minicluster-data/ts-2/wal/instance
I20260619 18:48:24.054585 4045 external_mini_cluster.cc:1428] Setting key 056232b24a98e85273630e7d4f7d096b
Jun 19 18:48:24 dist-test-slave-m9wg krb5kdc[14334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894903, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.67@KRBTEST.COM for kudu/127.3.243.126@KRBTEST.COM
I20260619 18:48:24.073004 14570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:48:24.055329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.67:48799 (local address 127.3.243.126:37933)
0619 18:48:24.055493 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 18:48:24.055497 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:48:24.056858 (+ 1361us) server_negotiation.cc:407] Connection header received
0619 18:48:24.058811 (+ 1953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:48:24.058817 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:48:24.058905 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:48:24.058987 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:48:24.060566 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:24.061292 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:24.062887 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:24.063293 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:24.063418 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:24.066915 (+ 3497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:48:24.066920 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:48:24.066923 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:48:24.066960 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:48:24.069984 (+ 3024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:48:24.070844 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:48:24.070848 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:48:24.070850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:48:24.070980 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:48:24.071345 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:48:24.071348 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:48:24.071351 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:48:24.071625 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:48:24.071732 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 18:48:24.072556 (+ 824us) server_negotiation.cc:299] Negotiation successful
0619 18:48:24.072726 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 18:48:24.073976 14837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:48:24.055920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:37933 (local address 127.3.243.67:48799)
0619 18:48:24.056714 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0619 18:48:24.056758 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 18:48:24.058550 (+ 1792us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:48:24.059234 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:48:24.059242 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:48:24.059760 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:48:24.060399 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:24.060423 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:24.061451 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:24.061455 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:24.062737 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:24.062749 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:24.063528 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:24.063535 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:24.063777 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:24.064501 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:48:24.064531 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:48:24.066687 (+ 2156us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:48:24.070141 (+ 3454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:48:24.070151 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:48:24.070167 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:48:24.070672 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:48:24.071063 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:48:24.071067 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:48:24.071069 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:48:24.071258 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:48:24.071767 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:48:24.071774 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:48:24.072110 (+ 336us) client_negotiation.cc:769] Sending connection context
0619 18:48:24.072439 (+ 329us) client_negotiation.cc:240] Negotiation successful
0619 18:48:24.072875 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":304,"threads_started":1}
I20260619 18:48:24.075325 14835 heartbeater.cc:347] Connected to a master server at 127.3.243.126:37933
I20260619 18:48:24.075608 14835 heartbeater.cc:464] Registering TS with master...
I20260619 18:48:24.076411 14835 heartbeater.cc:511] Master 127.3.243.126:37933 requested a full tablet report, sending...
I20260619 18:48:24.077747 14379 ts_manager.cc:194] Registered new tserver with Master: 877dcbffd6334d24afdd996375f7becc (127.3.243.67:40497)
I20260619 18:48:24.078567 14379 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.67@KRBTEST.COM'} at 127.3.243.67:48799
I20260619 18:48:24.082540 4045 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 18:48:24.099018 14570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:48:24.086841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45292 (local address 127.3.243.126:37933)
0619 18:48:24.087046 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 18:48:24.087053 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:48:24.087432 (+ 379us) server_negotiation.cc:407] Connection header received
0619 18:48:24.087642 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:48:24.087646 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:48:24.087761 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:48:24.087847 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:48:24.089297 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:24.090022 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:24.091186 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:24.091601 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:24.091697 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:24.094060 (+ 2363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:48:24.094065 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:48:24.094068 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:48:24.094098 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:48:24.096600 (+ 2502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:48:24.097217 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:48:24.097220 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:48:24.097222 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:48:24.097357 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:48:24.097662 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:48:24.097665 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:48:24.097667 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:48:24.097977 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:48:24.098178 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0619 18:48:24.098513 (+ 335us) server_negotiation.cc:299] Negotiation successful
0619 18:48:24.098717 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 18:48:24.103431 14379 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45292:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:48:24.110464 14379 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:48:24.133313 14568 heartbeater.cc:503] Master 127.3.243.126:37933 was elected leader, sending a full tablet report...
I20260619 18:48:24.134701 14850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:48:24.123218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56808 (local address 127.3.243.66:34971)
0619 18:48:24.125109 (+ 1891us) server_negotiation.cc:206] Beginning negotiation
0619 18:48:24.125118 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:48:24.125151 (+ 33us) server_negotiation.cc:407] Connection header received
0619 18:48:24.125258 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:48:24.125267 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:48:24.125507 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:48:24.126522 (+ 1015us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:48:24.126745 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:24.128462 (+ 1717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:24.131775 (+ 3313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:24.133297 (+ 1522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:24.133461 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:24.133660 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0619 18:48:24.134237 (+ 577us) server_negotiation.cc:299] Negotiation successful
0619 18:48:24.134407 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1750,"thread_start_us":1660,"threads_started":1}
I20260619 18:48:24.135128 14848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:48:24.122589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.67:40497 (local address 127.0.0.1:53414)
0619 18:48:24.123335 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0619 18:48:24.123378 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 18:48:24.123562 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:48:24.125545 (+ 1983us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:48:24.125555 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:48:24.125589 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:48:24.126126 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:24.126140 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:24.128617 (+ 2477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:24.128624 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:24.131086 (+ 2462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:24.131102 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:24.133972 (+ 2870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:24.133979 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:24.134157 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:24.134190 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 18:48:24.134276 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 18:48:24.134432 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"mutex_wait_us":2,"thread_start_us":948,"threads_started":1}
I20260619 18:48:24.135655 14847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:48:24.122647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53414 (local address 127.3.243.67:40497)
0619 18:48:24.124830 (+ 2183us) server_negotiation.cc:206] Beginning negotiation
0619 18:48:24.124839 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:48:24.124875 (+ 36us) server_negotiation.cc:407] Connection header received
0619 18:48:24.124975 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:48:24.124982 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:48:24.125284 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:48:24.126386 (+ 1102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:48:24.126681 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:24.128443 (+ 1762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:24.131300 (+ 2857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:24.132781 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:24.132904 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:24.135198 (+ 2294us) server_negotiation.cc:1035] Waiting for connection context
0619 18:48:24.135280 (+ 82us) server_negotiation.cc:299] Negotiation successful
0619 18:48:24.135439 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2006,"thread_start_us":1866,"threads_started":1}
I20260619 18:48:24.135943 14849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:48:24.122621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.66:34971 (local address 127.0.0.1:56808)
0619 18:48:24.124129 (+ 1508us) negotiation.cc:107] Waiting for socket to connect
0619 18:48:24.124145 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 18:48:24.124276 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:48:24.125743 (+ 1467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:48:24.125749 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:48:24.125781 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:48:24.126291 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:24.126303 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:24.128631 (+ 2328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:24.128637 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:24.131620 (+ 2983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:24.131632 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:24.133512 (+ 1880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:24.133518 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:24.133722 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:24.133788 (+ 66us) client_negotiation.cc:769] Sending connection context
0619 18:48:24.133934 (+ 146us) client_negotiation.cc:240] Negotiation successful
0619 18:48:24.135570 (+ 1636us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1399,"mutex_wait_us":1448,"thread_start_us":1263,"threads_started":1}
I20260619 18:48:24.138685 14851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:48:24.124428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.65:39555 (local address 127.0.0.1:48682)
0619 18:48:24.129203 (+ 4775us) negotiation.cc:107] Waiting for socket to connect
0619 18:48:24.129221 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 18:48:24.129332 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:48:24.131148 (+ 1816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:48:24.131153 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:48:24.131182 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:48:24.131637 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:24.131648 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:24.133697 (+ 2049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:24.133702 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:24.136369 (+ 2667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:24.136383 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:24.137878 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:24.137885 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:24.138045 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:24.138076 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 18:48:24.138163 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 18:48:24.138298 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4600,"thread_start_us":4447,"threads_started":1}
I20260619 18:48:24.138697 14852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:48:24.124986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48682 (local address 127.3.243.65:39555)
0619 18:48:24.130541 (+ 5555us) server_negotiation.cc:206] Beginning negotiation
0619 18:48:24.130550 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:48:24.130579 (+ 29us) server_negotiation.cc:407] Connection header received
0619 18:48:24.130676 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:48:24.130684 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:48:24.130945 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:48:24.131212 (+ 267us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:48:24.131831 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:24.133533 (+ 1702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:24.136546 (+ 3013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:24.137652 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:24.137784 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:24.138004 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0619 18:48:24.138219 (+ 215us) server_negotiation.cc:299] Negotiation successful
0619 18:48:24.138381 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5381,"thread_start_us":5225,"threads_started":1}
I20260619 18:48:24.139657 14637 tablet_service.cc:1467] Processing CreateTablet for tablet 72aae9355e554c57a001f789d941aa7b (DEFAULT_TABLE table=test-table [id=4110afbda34449d0a62ac2961ee18c77]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:48:24.139655 14770 tablet_service.cc:1467] Processing CreateTablet for tablet 72aae9355e554c57a001f789d941aa7b (DEFAULT_TABLE table=test-table [id=4110afbda34449d0a62ac2961ee18c77]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:48:24.141176 14770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72aae9355e554c57a001f789d941aa7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:48:24.141176 14637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72aae9355e554c57a001f789d941aa7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:48:24.142290 14503 tablet_service.cc:1467] Processing CreateTablet for tablet 72aae9355e554c57a001f789d941aa7b (DEFAULT_TABLE table=test-table [id=4110afbda34449d0a62ac2961ee18c77]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:48:24.143433 14503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72aae9355e554c57a001f789d941aa7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:48:24.154525 14853 tablet_bootstrap.cc:492] T 72aae9355e554c57a001f789d941aa7b P dae9c739628c495bb99a074e73e5bffe: Bootstrap starting.
I20260619 18:48:24.156895 14855 tablet_bootstrap.cc:492] T 72aae9355e554c57a001f789d941aa7b P 877dcbffd6334d24afdd996375f7becc: Bootstrap starting.
I20260619 18:48:24.158356 14854 tablet_bootstrap.cc:492] T 72aae9355e554c57a001f789d941aa7b P 7794eff267c147beaa8a5ba7458bd4ed: Bootstrap starting.
I20260619 18:48:24.159019 14853 tablet_bootstrap.cc:654] T 72aae9355e554c57a001f789d941aa7b P dae9c739628c495bb99a074e73e5bffe: Neither blocks nor log segments found. Creating new log.
I20260619 18:48:24.160290 14855 tablet_bootstrap.cc:654] T 72aae9355e554c57a001f789d941aa7b P 877dcbffd6334d24afdd996375f7becc: Neither blocks nor log segments found. Creating new log.
I20260619 18:48:24.160437 14853 log.cc:824] T 72aae9355e554c57a001f789d941aa7b P dae9c739628c495bb99a074e73e5bffe: Log is configured to *not* fsync() on all Append() calls
I20260619 18:48:24.161384 14855 log.cc:824] T 72aae9355e554c57a001f789d941aa7b P 877dcbffd6334d24afdd996375f7becc: Log is configured to *not* fsync() on all Append() calls
I20260619 18:48:24.161612 14854 tablet_bootstrap.cc:654] T 72aae9355e554c57a001f789d941aa7b P 7794eff267c147beaa8a5ba7458bd4ed: Neither blocks nor log segments found. Creating new log.
I20260619 18:48:24.162741 14854 log.cc:824] T 72aae9355e554c57a001f789d941aa7b P 7794eff267c147beaa8a5ba7458bd4ed: Log is configured to *not* fsync() on all Append() calls
I20260619 18:48:24.163617 14853 tablet_bootstrap.cc:492] T 72aae9355e554c57a001f789d941aa7b P dae9c739628c495bb99a074e73e5bffe: No bootstrap required, opened a new log
I20260619 18:48:24.163853 14853 ts_tablet_manager.cc:1397] T 72aae9355e554c57a001f789d941aa7b P dae9c739628c495bb99a074e73e5bffe: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260619 18:48:24.164112 14855 tablet_bootstrap.cc:492] T 72aae9355e554c57a001f789d941aa7b P 877dcbffd6334d24afdd996375f7becc: No bootstrap required, opened a new log
I20260619 18:48:24.164413 14855 ts_tablet_manager.cc:1397] T 72aae9355e554c57a001f789d941aa7b P 877dcbffd6334d24afdd996375f7becc: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260619 18:48:24.165795 14854 tablet_bootstrap.cc:492] T 72aae9355e554c57a001f789d941aa7b P 7794eff267c147beaa8a5ba7458bd4ed: No bootstrap required, opened a new log
I20260619 18:48:24.166056 14854 ts_tablet_manager.cc:1397] T 72aae9355e554c57a001f789d941aa7b P 7794eff267c147beaa8a5ba7458bd4ed: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260619 18:48:24.171756 14855 raft_consensus.cc:348] T 72aae9355e554c57a001f789d941aa7b P 877dcbffd6334d24afdd996375f7becc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "877dcbffd6334d24afdd996375f7becc" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40497 } } peers { permanent_uuid: "7794eff267c147beaa8a5ba7458bd4ed" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 34971 } } peers { permanent_uuid: "dae9c739628c495bb99a074e73e5bffe" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 39555 } }
I20260619 18:48:24.172112 14855 raft_consensus.cc:374] T 72aae9355e554c57a001f789d941aa7b P 877dcbffd6334d24afdd996375f7becc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:48:24.172233 14855 raft_consensus.cc:729] T 72aae9355e554c57a001f789d941aa7b P 877dcbffd6334d24afdd996375f7becc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 877dcbffd6334d24afdd996375f7becc, State: Initialized, Role: FOLLOWER
I20260619 18:48:24.172827 14855 consensus_queue.cc:260] T 72aae9355e554c57a001f789d941aa7b P 877dcbffd6334d24afdd996375f7becc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "877dcbffd6334d24afdd996375f7becc" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40497 } } peers { permanent_uuid: "7794eff267c147beaa8a5ba7458bd4ed" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 34971 } } peers { permanent_uuid: "dae9c739628c495bb99a074e73e5bffe" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 39555 } }
I20260619 18:48:24.173856 14835 heartbeater.cc:503] Master 127.3.243.126:37933 was elected leader, sending a full tablet report...
I20260619 18:48:24.174163 14853 raft_consensus.cc:348] T 72aae9355e554c57a001f789d941aa7b P dae9c739628c495bb99a074e73e5bffe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "877dcbffd6334d24afdd996375f7becc" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40497 } } peers { permanent_uuid: "7794eff267c147beaa8a5ba7458bd4ed" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 34971 } } peers { permanent_uuid: "dae9c739628c495bb99a074e73e5bffe" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 39555 } }
I20260619 18:48:24.174584 14853 raft_consensus.cc:374] T 72aae9355e554c57a001f789d941aa7b P dae9c739628c495bb99a074e73e5bffe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:48:24.174584 14855 ts_tablet_manager.cc:1428] T 72aae9355e554c57a001f789d941aa7b P 877dcbffd6334d24afdd996375f7becc: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20260619 18:48:24.174666 14853 raft_consensus.cc:729] T 72aae9355e554c57a001f789d941aa7b P dae9c739628c495bb99a074e73e5bffe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dae9c739628c495bb99a074e73e5bffe, State: Initialized, Role: FOLLOWER
I20260619 18:48:24.175246 14853 consensus_queue.cc:260] T 72aae9355e554c57a001f789d941aa7b P dae9c739628c495bb99a074e73e5bffe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "877dcbffd6334d24afdd996375f7becc" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40497 } } peers { permanent_uuid: "7794eff267c147beaa8a5ba7458bd4ed" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 34971 } } peers { permanent_uuid: "dae9c739628c495bb99a074e73e5bffe" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 39555 } }
I20260619 18:48:24.175966 14854 raft_consensus.cc:348] T 72aae9355e554c57a001f789d941aa7b P 7794eff267c147beaa8a5ba7458bd4ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "877dcbffd6334d24afdd996375f7becc" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40497 } } peers { permanent_uuid: "7794eff267c147beaa8a5ba7458bd4ed" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 34971 } } peers { permanent_uuid: "dae9c739628c495bb99a074e73e5bffe" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 39555 } }
I20260619 18:48:24.176396 14854 raft_consensus.cc:374] T 72aae9355e554c57a001f789d941aa7b P 7794eff267c147beaa8a5ba7458bd4ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:48:24.176483 14854 raft_consensus.cc:729] T 72aae9355e554c57a001f789d941aa7b P 7794eff267c147beaa8a5ba7458bd4ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7794eff267c147beaa8a5ba7458bd4ed, State: Initialized, Role: FOLLOWER
I20260619 18:48:24.176999 14853 ts_tablet_manager.cc:1428] T 72aae9355e554c57a001f789d941aa7b P dae9c739628c495bb99a074e73e5bffe: Time spent starting tablet: real 0.013s user 0.009s sys 0.005s
I20260619 18:48:24.177116 14854 consensus_queue.cc:260] T 72aae9355e554c57a001f789d941aa7b P 7794eff267c147beaa8a5ba7458bd4ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "877dcbffd6334d24afdd996375f7becc" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40497 } } peers { permanent_uuid: "7794eff267c147beaa8a5ba7458bd4ed" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 34971 } } peers { permanent_uuid: "dae9c739628c495bb99a074e73e5bffe" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 39555 } }
I20260619 18:48:24.178728 14854 ts_tablet_manager.cc:1428] T 72aae9355e554c57a001f789d941aa7b P 7794eff267c147beaa8a5ba7458bd4ed: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260619 18:48:24.179692 14702 heartbeater.cc:503] Master 127.3.243.126:37933 was elected leader, sending a full tablet report...
I20260619 18:48:24.227360 14859 raft_consensus.cc:482] T 72aae9355e554c57a001f789d941aa7b P 877dcbffd6334d24afdd996375f7becc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:48:24.227658 14859 raft_consensus.cc:504] T 72aae9355e554c57a001f789d941aa7b P 877dcbffd6334d24afdd996375f7becc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "877dcbffd6334d24afdd996375f7becc" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40497 } } peers { permanent_uuid: "7794eff267c147beaa8a5ba7458bd4ed" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 34971 } } peers { permanent_uuid: "dae9c739628c495bb99a074e73e5bffe" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 39555 } }
I20260619 18:48:24.229471 14859 leader_election.cc:288] T 72aae9355e554c57a001f789d941aa7b P 877dcbffd6334d24afdd996375f7becc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7794eff267c147beaa8a5ba7458bd4ed (127.3.243.66:34971), dae9c739628c495bb99a074e73e5bffe (127.3.243.65:39555)
I20260619 18:48:24.235075 14850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:48:24.229997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.67:36045 (local address 127.3.243.66:34971)
0619 18:48:24.230212 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0619 18:48:24.230218 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:48:24.230239 (+ 21us) server_negotiation.cc:407] Connection header received
0619 18:48:24.230313 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:48:24.230318 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:48:24.230391 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:48:24.230482 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:48:24.231040 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:24.231806 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:24.233495 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:24.234241 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:24.234396 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:24.234466 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 18:48:24.234716 (+ 250us) server_negotiation.cc:299] Negotiation successful
0619 18:48:24.234837 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 18:48:24.235096 14837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:48:24.229818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.66:34971 (local address 127.3.243.67:36045)
0619 18:48:24.230032 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0619 18:48:24.230052 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 18:48:24.230169 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:48:24.230665 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:48:24.230669 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:48:24.230700 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:48:24.230900 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:24.230909 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:24.231930 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:24.231935 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:24.233341 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:24.233356 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:24.234390 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:24.234396 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:24.234567 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:24.234590 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 18:48:24.234674 (+ 84us) client_negotiation.cc:240] Negotiation successful
0619 18:48:24.234806 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60,"mutex_wait_us":31}
I20260619 18:48:24.235823 14852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:48:24.230609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.67:40847 (local address 127.3.243.65:39555)
0619 18:48:24.230827 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0619 18:48:24.230832 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:48:24.230940 (+ 108us) server_negotiation.cc:407] Connection header received
0619 18:48:24.231096 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:48:24.231100 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:48:24.231197 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:48:24.231271 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:48:24.231720 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:24.232523 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:24.234170 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:24.234923 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:24.235060 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:24.235127 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 18:48:24.235478 (+ 351us) server_negotiation.cc:299] Negotiation successful
0619 18:48:24.235603 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260619 18:48:24.236177 14657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72aae9355e554c57a001f789d941aa7b" candidate_uuid: "877dcbffd6334d24afdd996375f7becc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7794eff267c147beaa8a5ba7458bd4ed" is_pre_election: true
I20260619 18:48:24.236372 14862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:48:24.230171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.65:39555 (local address 127.3.243.67:40847)
0619 18:48:24.230859 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0619 18:48:24.230874 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 18:48:24.230982 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:48:24.231299 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:48:24.231303 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:48:24.231326 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:48:24.231583 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:24.231590 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:24.232651 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:24.232656 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:24.234021 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:48:24.234032 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:48:24.235109 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:48:24.235115 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:48:24.235300 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:24.235325 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 18:48:24.235989 (+ 664us) client_negotiation.cc:240] Negotiation successful
0619 18:48:24.236083 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"mutex_wait_us":31,"thread_start_us":396,"threads_started":1}
I20260619 18:48:24.236675 14657 raft_consensus.cc:2454] T 72aae9355e554c57a001f789d941aa7b P 7794eff267c147beaa8a5ba7458bd4ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 877dcbffd6334d24afdd996375f7becc in term 0.
I20260619 18:48:24.237308 14726 leader_election.cc:302] T 72aae9355e554c57a001f789d941aa7b P 877dcbffd6334d24afdd996375f7becc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7794eff267c147beaa8a5ba7458bd4ed, 877dcbffd6334d24afdd996375f7becc; no voters:
I20260619 18:48:24.237368 14523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72aae9355e554c57a001f789d941aa7b" candidate_uuid: "877dcbffd6334d24afdd996375f7becc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dae9c739628c495bb99a074e73e5bffe" is_pre_election: true
I20260619 18:48:24.237641 14859 raft_consensus.cc:2781] T 72aae9355e554c57a001f789d941aa7b P 877dcbffd6334d24afdd996375f7becc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:48:24.237787 14859 raft_consensus.cc:482] T 72aae9355e554c57a001f789d941aa7b P 877dcbffd6334d24afdd996375f7becc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:48:24.237892 14859 raft_consensus.cc:3037] T 72aae9355e554c57a001f789d941aa7b P 877dcbffd6334d24afdd996375f7becc [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:48:24.237906 14523 raft_consensus.cc:2454] T 72aae9355e554c57a001f789d941aa7b P dae9c739628c495bb99a074e73e5bffe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 877dcbffd6334d24afdd996375f7becc in term 0.
I20260619 18:48:24.240305 14859 raft_consensus.cc:504] T 72aae9355e554c57a001f789d941aa7b P 877dcbffd6334d24afdd996375f7becc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "877dcbffd6334d24afdd996375f7becc" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40497 } } peers { permanent_uuid: "7794eff267c147beaa8a5ba7458bd4ed" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 34971 } } peers { permanent_uuid: "dae9c739628c495bb99a074e73e5bffe" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 39555 } }
I20260619 18:48:24.240885 14859 leader_election.cc:288] T 72aae9355e554c57a001f789d941aa7b P 877dcbffd6334d24afdd996375f7becc [CANDIDATE]: Term 1 election: Requested vote from peers 7794eff267c147beaa8a5ba7458bd4ed (127.3.243.66:34971), dae9c739628c495bb99a074e73e5bffe (127.3.243.65:39555)
I20260619 18:48:24.241251 14657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72aae9355e554c57a001f789d941aa7b" candidate_uuid: "877dcbffd6334d24afdd996375f7becc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7794eff267c147beaa8a5ba7458bd4ed"
I20260619 18:48:24.241315 14523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72aae9355e554c57a001f789d941aa7b" candidate_uuid: "877dcbffd6334d24afdd996375f7becc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dae9c739628c495bb99a074e73e5bffe"
I20260619 18:48:24.241430 14657 raft_consensus.cc:3037] T 72aae9355e554c57a001f789d941aa7b P 7794eff267c147beaa8a5ba7458bd4ed [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:48:24.241496 14523 raft_consensus.cc:3037] T 72aae9355e554c57a001f789d941aa7b P dae9c739628c495bb99a074e73e5bffe [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:48:24.243633 14657 raft_consensus.cc:2454] T 72aae9355e554c57a001f789d941aa7b P 7794eff267c147beaa8a5ba7458bd4ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 877dcbffd6334d24afdd996375f7becc in term 1.
I20260619 18:48:24.243750 14523 raft_consensus.cc:2454] T 72aae9355e554c57a001f789d941aa7b P dae9c739628c495bb99a074e73e5bffe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 877dcbffd6334d24afdd996375f7becc in term 1.
I20260619 18:48:24.243994 14726 leader_election.cc:302] T 72aae9355e554c57a001f789d941aa7b P 877dcbffd6334d24afdd996375f7becc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7794eff267c147beaa8a5ba7458bd4ed, 877dcbffd6334d24afdd996375f7becc; no voters:
I20260619 18:48:24.244263 14859 raft_consensus.cc:2781] T 72aae9355e554c57a001f789d941aa7b P 877dcbffd6334d24afdd996375f7becc [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:48:24.244674 14859 raft_consensus.cc:686] T 72aae9355e554c57a001f789d941aa7b P 877dcbffd6334d24afdd996375f7becc [term 1 LEADER]: Becoming Leader. State: Replica: 877dcbffd6334d24afdd996375f7becc, State: Running, Role: LEADER
I20260619 18:48:24.245128 14859 consensus_queue.cc:237] T 72aae9355e554c57a001f789d941aa7b P 877dcbffd6334d24afdd996375f7becc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "877dcbffd6334d24afdd996375f7becc" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40497 } } peers { permanent_uuid: "7794eff267c147beaa8a5ba7458bd4ed" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 34971 } } peers { permanent_uuid: "dae9c739628c495bb99a074e73e5bffe" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 39555 } }
I20260619 18:48:24.249384 14378 catalog_manager.cc:5495] T 72aae9355e554c57a001f789d941aa7b P 877dcbffd6334d24afdd996375f7becc reported cstate change: term changed from 0 to 1, leader changed from <none> to 877dcbffd6334d24afdd996375f7becc (127.3.243.67). New cstate: current_term: 1 leader_uuid: "877dcbffd6334d24afdd996375f7becc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "877dcbffd6334d24afdd996375f7becc" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40497 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7794eff267c147beaa8a5ba7458bd4ed" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 34971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dae9c739628c495bb99a074e73e5bffe" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 39555 } health_report { overall_health: UNKNOWN } } }
I20260619 18:48:24.294332 14847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:48:24.289402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53420 (local address 127.3.243.67:40497)
0619 18:48:24.289576 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 18:48:24.289580 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:48:24.289597 (+ 17us) server_negotiation.cc:407] Connection header received
0619 18:48:24.289726 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:48:24.289730 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:48:24.289840 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:48:24.289965 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:48:24.290258 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:24.290989 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:24.292249 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:24.292645 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:24.292748 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:24.293252 (+ 504us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:48:24.293824 (+ 572us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:48:24.293951 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 18:48:24.294051 (+ 100us) server_negotiation.cc:299] Negotiation successful
0619 18:48:24.294123 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 18:48:24.302578 14523 raft_consensus.cc:1261] T 72aae9355e554c57a001f789d941aa7b P dae9c739628c495bb99a074e73e5bffe [term 1 FOLLOWER]: Refusing update from remote peer 877dcbffd6334d24afdd996375f7becc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:48:24.302582 14657 raft_consensus.cc:1261] T 72aae9355e554c57a001f789d941aa7b P 7794eff267c147beaa8a5ba7458bd4ed [term 1 FOLLOWER]: Refusing update from remote peer 877dcbffd6334d24afdd996375f7becc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:48:24.303511 14863 consensus_queue.cc:1035] T 72aae9355e554c57a001f789d941aa7b P 877dcbffd6334d24afdd996375f7becc [LEADER]: Connected to new peer: Peer: permanent_uuid: "dae9c739628c495bb99a074e73e5bffe" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 39555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:48:24.303856 14859 consensus_queue.cc:1035] T 72aae9355e554c57a001f789d941aa7b P 877dcbffd6334d24afdd996375f7becc [LEADER]: Connected to new peer: Peer: permanent_uuid: "7794eff267c147beaa8a5ba7458bd4ed" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 34971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 18:48:24.304277 14836 tablet.cc:2369] T 72aae9355e554c57a001f789d941aa7b P 877dcbffd6334d24afdd996375f7becc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:48:24.316601 14867 mvcc.cc:205] Tried to move back new op lower bound from 7298641528013336576 to 7298641527791685632. Current Snapshot: MvccSnapshot[applied={T|T < 7298641528013336576}]
I20260619 18:48:24.327327 14378 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45292:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:48:24.327569 14378 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45292:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:48:24.331017 14378 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7533f699512645be834b42093f86d2e4: Sending DeleteTablet for 3 replicas of tablet 72aae9355e554c57a001f789d941aa7b
I20260619 18:48:24.332193 14770 tablet_service.cc:1514] Processing DeleteTablet for tablet 72aae9355e554c57a001f789d941aa7b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:48:24 UTC) from {username='kudu', principal='kudu/127.3.243.126@KRBTEST.COM'} at 127.0.0.1:53414
I20260619 18:48:24.332270 14637 tablet_service.cc:1514] Processing DeleteTablet for tablet 72aae9355e554c57a001f789d941aa7b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:48:24 UTC) from {username='kudu', principal='kudu/127.3.243.126@KRBTEST.COM'} at 127.0.0.1:56808
I20260619 18:48:24.332389 14503 tablet_service.cc:1514] Processing DeleteTablet for tablet 72aae9355e554c57a001f789d941aa7b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:48:24 UTC) from {username='kudu', principal='kudu/127.3.243.126@KRBTEST.COM'} at 127.0.0.1:48682
I20260619 18:48:24.334478 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 14440
I20260619 18:48:24.360322 14876 tablet_replica.cc:330] T 72aae9355e554c57a001f789d941aa7b P 7794eff267c147beaa8a5ba7458bd4ed: stopping tablet replica
I20260619 18:48:24.360739 14876 raft_consensus.cc:2229] T 72aae9355e554c57a001f789d941aa7b P 7794eff267c147beaa8a5ba7458bd4ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:48:24.361219 14876 raft_consensus.cc:2258] T 72aae9355e554c57a001f789d941aa7b P 7794eff267c147beaa8a5ba7458bd4ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:48:24.363282 14877 tablet_replica.cc:330] T 72aae9355e554c57a001f789d941aa7b P dae9c739628c495bb99a074e73e5bffe: stopping tablet replica
I20260619 18:48:24.363729 14877 raft_consensus.cc:2229] T 72aae9355e554c57a001f789d941aa7b P dae9c739628c495bb99a074e73e5bffe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:48:24.364274 14877 raft_consensus.cc:2258] T 72aae9355e554c57a001f789d941aa7b P dae9c739628c495bb99a074e73e5bffe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:48:24.366537 14877 ts_tablet_manager.cc:1905] T 72aae9355e554c57a001f789d941aa7b P dae9c739628c495bb99a074e73e5bffe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:48:24.367038 14876 ts_tablet_manager.cc:1905] T 72aae9355e554c57a001f789d941aa7b P 7794eff267c147beaa8a5ba7458bd4ed: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 19 18:48:24 dist-test-slave-m9wg krb5kdc[14334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894902, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.65@KRBTEST.COM
I20260619 18:48:24.373195 14877 ts_tablet_manager.cc:1918] T 72aae9355e554c57a001f789d941aa7b P dae9c739628c495bb99a074e73e5bffe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:48:24.373359 14877 log.cc:1196] T 72aae9355e554c57a001f789d941aa7b P dae9c739628c495bb99a074e73e5bffe: Deleting WAL directory at /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894810715282-4045-0/minicluster-data/ts-0/wal/wals/72aae9355e554c57a001f789d941aa7b
I20260619 18:48:24.373858 14877 ts_tablet_manager.cc:1939] T 72aae9355e554c57a001f789d941aa7b P dae9c739628c495bb99a074e73e5bffe: Deleting consensus metadata
I20260619 18:48:24.374256 14876 ts_tablet_manager.cc:1918] T 72aae9355e554c57a001f789d941aa7b P 7794eff267c147beaa8a5ba7458bd4ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:48:24.374409 14876 log.cc:1196] T 72aae9355e554c57a001f789d941aa7b P 7794eff267c147beaa8a5ba7458bd4ed: Deleting WAL directory at /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894810715282-4045-0/minicluster-data/ts-1/wal/wals/72aae9355e554c57a001f789d941aa7b
I20260619 18:48:24.374923 14876 ts_tablet_manager.cc:1939] T 72aae9355e554c57a001f789d941aa7b P 7794eff267c147beaa8a5ba7458bd4ed: Deleting consensus metadata
I20260619 18:48:24.375810 14366 catalog_manager.cc:4841] TS dae9c739628c495bb99a074e73e5bffe (127.3.243.65:39555): tablet 72aae9355e554c57a001f789d941aa7b (table test-table [id=4110afbda34449d0a62ac2961ee18c77]) successfully deleted
I20260619 18:48:24.376667 14875 tablet_replica.cc:330] T 72aae9355e554c57a001f789d941aa7b P 877dcbffd6334d24afdd996375f7becc: stopping tablet replica
I20260619 18:48:24.377133 14875 raft_consensus.cc:2229] T 72aae9355e554c57a001f789d941aa7b P 877dcbffd6334d24afdd996375f7becc [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:48:24.377707 14875 raft_consensus.cc:2258] T 72aae9355e554c57a001f789d941aa7b P 877dcbffd6334d24afdd996375f7becc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:48:24.378909 14852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:48:24.336002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48698 (local address 127.3.243.65:39555)
0619 18:48:24.336203 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 18:48:24.336210 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:48:24.358945 (+ 22735us) server_negotiation.cc:407] Connection header received
0619 18:48:24.359497 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:48:24.359505 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:48:24.359600 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:48:24.359678 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:48:24.362818 (+ 3140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:24.363988 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:24.365873 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:24.366457 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:24.366556 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:24.371815 (+ 5259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:48:24.371829 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:48:24.371851 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:48:24.371889 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:48:24.375481 (+ 3592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:48:24.376551 (+ 1070us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:48:24.376560 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:48:24.376563 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:48:24.376731 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:48:24.377196 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:48:24.377201 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:48:24.377205 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:48:24.377683 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:48:24.378003 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0619 18:48:24.378366 (+ 363us) server_negotiation.cc:299] Negotiation successful
0619 18:48:24.378570 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 18:48:24.379794 14563 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:48:24.380468 14875 ts_tablet_manager.cc:1905] T 72aae9355e554c57a001f789d941aa7b P 877dcbffd6334d24afdd996375f7becc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:48:24.386454 14875 ts_tablet_manager.cc:1918] T 72aae9355e554c57a001f789d941aa7b P 877dcbffd6334d24afdd996375f7becc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:48:24.386621 14875 log.cc:1196] T 72aae9355e554c57a001f789d941aa7b P 877dcbffd6334d24afdd996375f7becc: Deleting WAL directory at /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894810715282-4045-0/minicluster-data/ts-2/wal/wals/72aae9355e554c57a001f789d941aa7b
I20260619 18:48:24.387064 14875 ts_tablet_manager.cc:1939] T 72aae9355e554c57a001f789d941aa7b P 877dcbffd6334d24afdd996375f7becc: Deleting consensus metadata
I20260619 18:48:24.388593 14363 catalog_manager.cc:4841] TS 877dcbffd6334d24afdd996375f7becc (127.3.243.67:40497): tablet 72aae9355e554c57a001f789d941aa7b (table test-table [id=4110afbda34449d0a62ac2961ee18c77]) successfully deleted
I20260619 18:48:24.388931 14366 catalog_manager.cc:4841] TS 7794eff267c147beaa8a5ba7458bd4ed (127.3.243.66:34971): tablet 72aae9355e554c57a001f789d941aa7b (table test-table [id=4110afbda34449d0a62ac2961ee18c77]) successfully deleted
I20260619 18:48:24.525978 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 14440
I20260619 18:48:24.546950 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 14574
Jun 19 18:48:24 dist-test-slave-m9wg krb5kdc[14334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894902, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.66@KRBTEST.COM
I20260619 18:48:24.562680 14850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:48:24.547805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56814 (local address 127.3.243.66:34971)
0619 18:48:24.548014 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0619 18:48:24.548020 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:48:24.548038 (+ 18us) server_negotiation.cc:407] Connection header received
0619 18:48:24.548199 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:48:24.548205 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:48:24.548278 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:48:24.548364 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:48:24.549583 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:24.550332 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:24.551540 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:24.551942 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:24.552048 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:24.556440 (+ 4392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:48:24.556459 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:48:24.556481 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:48:24.556524 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:48:24.559989 (+ 3465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:48:24.560668 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:48:24.560678 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:48:24.560682 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:48:24.560833 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:48:24.561143 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:48:24.561147 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:48:24.561151 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:48:24.561625 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:48:24.561908 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0619 18:48:24.562085 (+ 177us) server_negotiation.cc:299] Negotiation successful
0619 18:48:24.562316 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260619 18:48:24.563367 14697 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:48:24.654036 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 14574
I20260619 18:48:24.668196 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 14707
Jun 19 18:48:24 dist-test-slave-m9wg krb5kdc[14334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894902, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.67@KRBTEST.COM
I20260619 18:48:24.685815 14847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:48:24.669073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53424 (local address 127.3.243.67:40497)
0619 18:48:24.669285 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 18:48:24.669302 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 18:48:24.669326 (+ 24us) server_negotiation.cc:407] Connection header received
0619 18:48:24.669393 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:48:24.669399 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:48:24.669503 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:48:24.669610 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:48:24.670816 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:24.672022 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:24.673338 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:48:24.673999 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:48:24.674135 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:48:24.678664 (+ 4529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:48:24.678681 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:48:24.678706 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:48:24.678757 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:48:24.682596 (+ 3839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:48:24.683324 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:48:24.683335 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:48:24.683338 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:48:24.683521 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:48:24.683954 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:48:24.683960 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:48:24.683963 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:48:24.684523 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:48:24.684836 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0619 18:48:24.685139 (+ 303us) server_negotiation.cc:299] Negotiation successful
0619 18:48:24.685384 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 18:48:24.686589 14830 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:48:24.805627 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 14707
I20260619 18:48:24.821084 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 14347
I20260619 18:48:24.822139 14409 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:48:24.931529 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 14347
2026-06-19T18:48:24Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4068 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:48:25 dist-test-slave-m9wg krb5kdc[14885](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:48:25 dist-test-slave-m9wg krb5kdc[14885](info): set up 2 sockets
Jun 19 18:48:25 dist-test-slave-m9wg krb5kdc[14885](info): commencing operation
krb5kdc: starting...
W20260619 18:48:25.928894 4045 mini_kdc.cc:121] Time spent starting KDC: real 0.960s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/krb5kdc/test-user.keytab.
Jun 19 18:48:26 dist-test-slave-m9wg krb5kdc[14885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:48:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:48:26Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.3.243.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.3.243.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.3.243.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/krb5kdc/rangeradmin_127.3.243.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.3.243.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/krb5kdc/rangeradmin_127.3.243.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.3.243.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.3.243.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.3.243.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/krb5kdc/rangerlookup_127.3.243.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.3.243.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/krb5kdc/rangerlookup_127.3.243.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.243.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.243.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.243.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/krb5kdc/HTTP_127.3.243.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.243.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/krb5kdc/HTTP_127.3.243.84@KRBTEST.COM.keytab.
I20260619 18:48:26.207958 4045 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskRgbpDZ/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/minicluster-data/postgres -l logfile start
2026-06-19 18:48:28.358 UTC [14915] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 18:48:28.358 UTC [14915] LOG: listening on IPv4 address "127.3.243.84", port 54815
2026-06-19 18:48:28.361 UTC [14915] LOG: listening on Unix socket "/tmp/.s.PGSQL.54815"
2026-06-19 18:48:28.380 UTC [14918] LOG: database system was shut down at 2026-06-19 18:48:27 UTC
2026-06-19 18:48:28.383 UTC [14915] LOG: database system is ready to accept connections
I20260619 18:48:29.272455 4045 mini_postgres.cc:96] Postgres bound to 54815
2026-06-19 18:48:29.288 UTC [14926] FATAL: database "slave" does not exist
127.3.243.84:54815 - accepting connections
I20260619 18:48:29.288990 4045 mini_ranger.cc:161] Starting Ranger...
I20260619 18:48:29.311364 4045 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 18:48:29.698846 4045 mini_ranger.cc:87] Created ranger Postgres database
I20260619 18:48:29.698983 4045 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/minicluster-data/ranger-admin
2026-06-19 18:48:29,969 [I] DB FLAVOR :POSTGRES
2026-06-19 18:48:29,970 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 18:48:29,970 [I] Checking connection..
2026-06-19 18:48:29,970 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRgbpDZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRgbpDZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.243.84:54815/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 18:48:30,207 [I] Checking connection passed.
2026-06-19 18:48:30,207 [I] --------- Verifying version history table ---------
2026-06-19 18:48:30,207 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRgbpDZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRgbpDZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.243.84:54815/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 18:48:30,451 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 18:48:30,451 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRgbpDZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRgbpDZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.243.84:54815/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 18:48:30,697 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 18:48:30,698 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 18:48:30,698 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRgbpDZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRgbpDZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.243.84:54815/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskRgbpDZ/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 18:48:30.926 UTC [14990] WARNING: there is no transaction in progress
2026-06-19 18:48:30,931 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 18:48:30,931 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRgbpDZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRgbpDZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.243.84:54815/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 18:48:31,176 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 18:48:31,176 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 18:48:31,176 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRgbpDZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRgbpDZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.243.84:54815/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 18:48:31,417 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRgbpDZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRgbpDZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.243.84:54815/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 18:48:31,665 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRgbpDZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRgbpDZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.243.84:54815/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-m9wg.c.gcp-upstream.internal','N') ;"
2026-06-19 18:48:31,903 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRgbpDZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRgbpDZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.243.84:54815/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 18:48:32,137 [I] Table x_portal_user does not exist in database ranger
2026-06-19 18:48:32,137 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRgbpDZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRgbpDZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.243.84:54815/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 18:48:32,382 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 18:48:32,382 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRgbpDZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRgbpDZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.243.84:54815/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 18:48:32,610 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRgbpDZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRgbpDZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.243.84:54815/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 18:48:32,867 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 18:48:32,867 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRgbpDZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRgbpDZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.243.84:54815/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskRgbpDZ/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 18:48:33.728 UTC [15098] WARNING: there is no transaction in progress
2026-06-19 18:48:33.743 UTC [15098] WARNING: there is no transaction in progress
2026-06-19 18:48:33.756 UTC [15098] WARNING: there is no transaction in progress
2026-06-19 18:48:33.770 UTC [15098] WARNING: there is no transaction in progress
2026-06-19 18:48:33.784 UTC [15098] WARNING: there is no transaction in progress
2026-06-19 18:48:33.800 UTC [15098] WARNING: there is no transaction in progress
2026-06-19 18:48:33.849 UTC [15098] WARNING: there is no transaction in progress
2026-06-19 18:48:33.855 UTC [15098] WARNING: there is no transaction in progress
2026-06-19 18:48:33.868 UTC [15098] WARNING: there is no transaction in progress
2026-06-19 18:48:33.877 UTC [15098] WARNING: there is no transaction in progress
2026-06-19 18:48:33.886 UTC [15098] WARNING: there is no transaction in progress
2026-06-19 18:48:33.899 UTC [15098] WARNING: there is no transaction in progress
2026-06-19 18:48:33.907 UTC [15098] WARNING: there is no transaction in progress
2026-06-19 18:48:33.913 UTC [15098] WARNING: there is no transaction in progress
2026-06-19 18:48:33.918 UTC [15098] WARNING: there is no transaction in progress
2026-06-19 18:48:33.923 UTC [15098] WARNING: there is no transaction in progress
2026-06-19 18:48:34,430 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 18:48:34,430 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRgbpDZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRgbpDZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.243.84:54815/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 18:48:34,651 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 18:48:34,651 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRgbpDZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRgbpDZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.243.84:54815/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 18:48:34,897 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 18:48:34,898 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRgbpDZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRgbpDZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.243.84:54815/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 18:48:35,143 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 18:48:35,144 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRgbpDZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRgbpDZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.243.84:54815/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 18:48:35,387 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRgbpDZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRgbpDZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.243.84:54815/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 18:48:35,609 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRgbpDZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRgbpDZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.243.84:54815/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-m9wg.c.gcp-upstream.internal';"
2026-06-19 18:48:35,829 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 18:48:35,829 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRgbpDZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRgbpDZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.243.84:54815/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 18:48:36,077 [I] DB_PATCHES have already been applied
I20260619 18:48:36.083103 4045 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/minicluster-data/ranger-admin:/tmp/dist-test-taskRgbpDZ/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskRgbpDZ/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskRgbpDZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRgbpDZ/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 18:48:36.083206 4045 mini_ranger.cc:193] Using host: 127.3.243.84
I20260619 18:48:36.097036 4045 mini_ranger.cc:227] Ranger admin URL: http://127.3.243.84:42705
Jun 19, 2026 6:48:36 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 6:48:36 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 6:48:36 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 6:48:36 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 6:48:36 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 6:48:36 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 6:48:36 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 18:48:37.095245 4045 mini_ranger.cc:160] Time spent starting Ranger: real 7.806s user 0.003s sys 0.032s
Jun 19, 2026 6:48:55 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 6:48:55 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 19, 2026 6:48:55 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 6:48:55 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 6:48:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 6:48:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 6:48:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 6:48:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 6:48:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 6:48:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 6:48:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 6:48:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 6:48:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 6:48:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 6:48:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 6:48:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 6:48:55 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 6:48:56 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 18:48:56.933218 4045 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.3.243.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.3.243.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.3.243.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/krb5kdc/rangerkms_127.3.243.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.3.243.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/krb5kdc/rangerkms_127.3.243.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.243.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.3.243.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.243.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/krb5kdc/HTTP_127.3.243.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.243.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/krb5kdc/HTTP_127.3.243.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 18:48:57 dist-test-slave-m9wg krb5kdc[14885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894937, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 18:48:57.103824 4045 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 18:48:57.122551 4045 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 18:48:57.833635 4045 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 18:48:57.833796 4045 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/minicluster-data/ranger-kms
I20260619 18:48:57.833936 4045 mini_ranger_kms.cc:227] Using postgres at 127.3.243.84:54815
2026-06-19 18:48:58,260 [I] DB FLAVOR :POSTGRES
2026-06-19 18:48:58,261 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 18:48:58,261 [I] Checking connection
2026-06-19 18:48:58,261 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRgbpDZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.243.84:54815/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 18:48:58,499 [I] connection success
2026-06-19 18:48:58,499 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 18:48:58,499 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 18:48:58,499 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRgbpDZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.243.84:54815/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 18:48:58,730 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 18:48:58,731 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 18:48:58,731 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 18:48:58,731 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRgbpDZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.243.84:54815/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 18:48:58,966 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 18:48:59.381655 4045 mini_ranger_kms.cc:310] Created kms service
I20260619 18:48:59.565697 4045 mini_ranger_kms.cc:326] Created kudu user
I20260619 18:48:59.606298 4045 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 18:48:59 dist-test-slave-m9wg krb5kdc[14885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894937, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.3.243.84@KRBTEST.COM
I20260619 18:48:59.956197 4045 mini_ranger_kms.cc:384] Added ranger policy
I20260619 18:48:59.956547 4045 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/minicluster-data/ranger-kms:/tmp/dist-test-taskRgbpDZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRgbpDZ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskRgbpDZ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 18:48:59.956606 4045 mini_ranger_kms.cc:240] Using host: 127.3.243.84
I20260619 18:48:59.972517 4045 mini_ranger_kms.cc:276] Ranger KMS PID: 15278
I20260619 18:48:59.972694 4045 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.3.243.84:38513
Jun 19, 2026 6:49:00 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 6:49:00 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 6:49:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 6:49:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 6:49:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 6:49:01 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260619 18:49:01.992635 4045 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.889s user 0.136s sys 0.307s
I20260619 18:49:01.992878 4045 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 18:49:01.993095 4045 mini_ranger_kms.cc:401] 127.3.243.84:38513/kms/v1/keys
Jun 19 18:49:02 dist-test-slave-m9wg krb5kdc[14885](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781894942, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.243.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 19 18:49:02 dist-test-slave-m9wg krb5kdc[14885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894942, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.243.84@KRBTEST.COM for HTTP/127.3.243.84@KRBTEST.COM
Jun 19 18:49:02 dist-test-slave-m9wg krb5kdc[14885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894942, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.243.84@KRBTEST.COM for HTTP/127.3.243.84@KRBTEST.COM
Jun 19 18:49:05 dist-test-slave-m9wg krb5kdc[14885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894945, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.3.243.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab.
Entry for principal kudu/127.3.243.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab.
I20260619 18:49:05.145056 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.126:40583
--webserver_interface=127.3.243.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab
--principal=kudu/127.3.243.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:39537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.126:40583
--ranger_config_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.243.84:38513/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:49:05.665498 15329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:49:05.665865 15329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:49:05.666056 15329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:49:05.675699 15329 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:49:05.675892 15329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:49:05.676007 15329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:49:05.676165 15329 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:49:05.676288 15329 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:49:05.689706 15329 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:39537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/minicluster-data/master-0/wal
--ranger_kms_url=127.3.243.84:38513/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.3.243.126:40583
--ranger_config_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab
--principal=kudu/127.3.243.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.126:40583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.243.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:49:05.691151 15329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:49:05.692651 15329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:49:05.706244 15334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:49:05.706369 15337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:49:05.706511 15335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:49:05.708027 15329 server_base.cc:1034] running on GCE node
I20260619 18:49:05.708786 15329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:49:05.710191 15329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:49:05.711396 15329 hybrid_clock.cc:648] HybridClock initialized: now 1781894945711356 us; error 25 us; skew 500 ppm
I20260619 18:49:05.711732 15329 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:49:05 dist-test-slave-m9wg krb5kdc[14885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894945, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:49:05.714859 15329 init.cc:377] Logged in from keytab as kudu/127.3.243.126@KRBTEST.COM (short username kudu)
I20260619 18:49:05.716744 15329 webserver.cc:460] Webserver started at http://127.3.243.126:33747/ using document root <none> and password file <none>
I20260619 18:49:05.717471 15329 fs_manager.cc:359] Metadata directory not provided
I20260619 18:49:05.717633 15329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:49:05.717957 15329 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 18:49:05 dist-test-slave-m9wg krb5kdc[14885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894945, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.126@KRBTEST.COM for HTTP/127.3.243.84@KRBTEST.COM
I20260619 18:49:05.771577 15329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/minicluster-data/master-0/data/instance:
uuid: "cdd11781508e44d5a36669725919104d"
format_stamp: "Formatted at 2026-06-19 18:49:05 on dist-test-slave-m9wg"
server_key: "d821fb5ed1dfc179c3c9ddc81ace836d"
server_key_iv: "f34e5f13e0563db2825264b3ad1891c1"
server_key_version: "kuduclusterkey@0"
I20260619 18:49:05.772614 15329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/minicluster-data/master-0/wal/instance:
uuid: "cdd11781508e44d5a36669725919104d"
format_stamp: "Formatted at 2026-06-19 18:49:05 on dist-test-slave-m9wg"
server_key: "d821fb5ed1dfc179c3c9ddc81ace836d"
server_key_iv: "f34e5f13e0563db2825264b3ad1891c1"
server_key_version: "kuduclusterkey@0"
I20260619 18:49:05.779733 15329 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.005s
I20260619 18:49:05.802774 15344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:49:05.804426 15329 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260619 18:49:05.804641 15329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/minicluster-data/master-0/wal
uuid: "cdd11781508e44d5a36669725919104d"
format_stamp: "Formatted at 2026-06-19 18:49:05 on dist-test-slave-m9wg"
server_key: "d821fb5ed1dfc179c3c9ddc81ace836d"
server_key_iv: "f34e5f13e0563db2825264b3ad1891c1"
server_key_version: "kuduclusterkey@0"
I20260619 18:49:05.804903 15329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:49:05.830308 15329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:49:05.835045 15329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:49:05.835498 15329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:49:05.871366 15329 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.126:40583
I20260619 18:49:05.871456 15395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.126:40583 every 8 connection(s)
I20260619 18:49:05.873029 15329 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
I20260619 18:49:05.877910 15396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:49:05.882941 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 15329
I20260619 18:49:05.883137 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/minicluster-data/master-0/wal/instance
I20260619 18:49:05.883893 4045 external_mini_cluster.cc:1428] Setting key f20bd174fbf5eb53e9e3f7e230e4a947
I20260619 18:49:05.892259 15396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdd11781508e44d5a36669725919104d: Bootstrap starting.
I20260619 18:49:05.896620 15396 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cdd11781508e44d5a36669725919104d: Neither blocks nor log segments found. Creating new log.
I20260619 18:49:05.898288 15396 log.cc:824] T 00000000000000000000000000000000 P cdd11781508e44d5a36669725919104d: Log is configured to *not* fsync() on all Append() calls
Jun 19 18:49:05 dist-test-slave-m9wg krb5kdc[14885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894945, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.126@KRBTEST.COM
I20260619 18:49:05.902485 15396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdd11781508e44d5a36669725919104d: No bootstrap required, opened a new log
I20260619 18:49:05.910384 15399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:49:05.886147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45700 (local address 127.3.243.126:40583)
0619 18:49:05.887414 (+ 1267us) server_negotiation.cc:206] Beginning negotiation
0619 18:49:05.887428 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 18:49:05.887485 (+ 57us) server_negotiation.cc:407] Connection header received
0619 18:49:05.890851 (+ 3366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:49:05.890892 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:49:05.891377 (+ 485us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:49:05.891795 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:49:05.893143 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:49:05.894851 (+ 1708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:49:05.896192 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:49:05.897126 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:49:05.897291 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:49:05.901881 (+ 4590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:49:05.901890 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:49:05.901919 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:49:05.901972 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:49:05.906014 (+ 4042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:49:05.906661 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:49:05.906667 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:49:05.906674 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:49:05.906838 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:49:05.907226 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:49:05.907230 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:49:05.907232 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:49:05.907844 (+ 612us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:49:05.908260 (+ 416us) server_negotiation.cc:1035] Waiting for connection context
0619 18:49:05.908687 (+ 427us) server_negotiation.cc:299] Negotiation successful
0619 18:49:05.909098 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1017,"thread_start_us":710,"threads_started":1}
I20260619 18:49:05.914412 15396 raft_consensus.cc:348] T 00000000000000000000000000000000 P cdd11781508e44d5a36669725919104d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdd11781508e44d5a36669725919104d" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 40583 } }
I20260619 18:49:05.914852 15396 raft_consensus.cc:374] T 00000000000000000000000000000000 P cdd11781508e44d5a36669725919104d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:49:05.914991 15396 raft_consensus.cc:729] T 00000000000000000000000000000000 P cdd11781508e44d5a36669725919104d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdd11781508e44d5a36669725919104d, State: Initialized, Role: FOLLOWER
I20260619 18:49:05.915640 15396 consensus_queue.cc:260] T 00000000000000000000000000000000 P cdd11781508e44d5a36669725919104d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdd11781508e44d5a36669725919104d" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 40583 } }
I20260619 18:49:05.915942 15396 raft_consensus.cc:388] T 00000000000000000000000000000000 P cdd11781508e44d5a36669725919104d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:49:05.916133 15396 raft_consensus.cc:482] T 00000000000000000000000000000000 P cdd11781508e44d5a36669725919104d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:49:05.916388 15396 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cdd11781508e44d5a36669725919104d [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:49:05.919642 15396 raft_consensus.cc:504] T 00000000000000000000000000000000 P cdd11781508e44d5a36669725919104d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdd11781508e44d5a36669725919104d" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 40583 } }
I20260619 18:49:05.920362 15396 leader_election.cc:302] T 00000000000000000000000000000000 P cdd11781508e44d5a36669725919104d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cdd11781508e44d5a36669725919104d; no voters:
I20260619 18:49:05.921311 15396 leader_election.cc:288] T 00000000000000000000000000000000 P cdd11781508e44d5a36669725919104d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:49:05.921576 15401 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cdd11781508e44d5a36669725919104d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:49:05.923184 15401 raft_consensus.cc:686] T 00000000000000000000000000000000 P cdd11781508e44d5a36669725919104d [term 1 LEADER]: Becoming Leader. State: Replica: cdd11781508e44d5a36669725919104d, State: Running, Role: LEADER
I20260619 18:49:05.923905 15401 consensus_queue.cc:237] T 00000000000000000000000000000000 P cdd11781508e44d5a36669725919104d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdd11781508e44d5a36669725919104d" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 40583 } }
I20260619 18:49:05.924067 15396 sys_catalog.cc:564] T 00000000000000000000000000000000 P cdd11781508e44d5a36669725919104d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:49:05.927192 15402 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdd11781508e44d5a36669725919104d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cdd11781508e44d5a36669725919104d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdd11781508e44d5a36669725919104d" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 40583 } } }
I20260619 18:49:05.927580 15403 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdd11781508e44d5a36669725919104d [sys.catalog]: SysCatalogTable state changed. Reason: New leader cdd11781508e44d5a36669725919104d. Latest consensus state: current_term: 1 leader_uuid: "cdd11781508e44d5a36669725919104d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdd11781508e44d5a36669725919104d" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 40583 } } }
I20260619 18:49:05.927935 15402 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdd11781508e44d5a36669725919104d [sys.catalog]: This master's current role is: LEADER
I20260619 18:49:05.928050 15403 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdd11781508e44d5a36669725919104d [sys.catalog]: This master's current role is: LEADER
I20260619 18:49:05.929314 15407 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:49:05.936642 15407 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:49:05.949262 15396 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 18:49:05.949371 15407 catalog_manager.cc:1347] Generated new cluster ID: 0171ebf2182a4382977035cbf4172a13
I20260619 18:49:05.949569 15407 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:49:05.997576 15407 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:49:05.999864 15407 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:49:06.010731 15407 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cdd11781508e44d5a36669725919104d: Generated new TSK 0
I20260619 18:49:06.011981 15407 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.243.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.65.keytab.
Entry for principal kudu/127.3.243.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.65.keytab.
I20260619 18:49:06.799997 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.243.84:38513/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.65:0
--local_ip_for_outbound_sockets=127.3.243.65
--webserver_interface=127.3.243.65
--webserver_port=0
--tserver_master_addrs=127.3.243.126:40583
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.65.keytab
--principal=kudu/127.3.243.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:39537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 18:49:07 dist-test-slave-m9wg krb5kdc[14885](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781894947, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260619 18:49:07.315318 15441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:49:07.315812 15441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:49:07.316077 15441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:49:07.326372 15441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:49:07.326627 15441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:49:07.326920 15441 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.65
I20260619 18:49:07.342765 15441 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:39537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.3.243.84:38513/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.65.keytab
--principal=kudu/127.3.243.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.243.65
--webserver_port=0
--tserver_master_addrs=127.3.243.126:40583
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.65
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:49:07.344437 15441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:49:07.346094 15441 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:49:07.366727 15451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:49:07.368204 15452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:49:07.374012 15441 server_base.cc:1034] running on GCE node
W20260619 18:49:07.374267 15454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:49:07.375151 15441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:49:07.376412 15441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:49:07.377703 15441 hybrid_clock.cc:648] HybridClock initialized: now 1781894947377570 us; error 93 us; skew 500 ppm
I20260619 18:49:07.378100 15441 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:49:07 dist-test-slave-m9wg krb5kdc[14885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894947, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:49:07.386876 15441 init.cc:377] Logged in from keytab as kudu/127.3.243.65@KRBTEST.COM (short username kudu)
I20260619 18:49:07.389644 15441 webserver.cc:460] Webserver started at http://127.3.243.65:38027/ using document root <none> and password file <none>
I20260619 18:49:07.390481 15441 fs_manager.cc:359] Metadata directory not provided
I20260619 18:49:07.390657 15441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:49:07.391072 15441 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 18:49:07 dist-test-slave-m9wg krb5kdc[14885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894947, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.65@KRBTEST.COM for HTTP/127.3.243.84@KRBTEST.COM
I20260619 18:49:07.414053 15441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/minicluster-data/ts-0/data/instance:
uuid: "0e48a2270b0f41a3a463e5aa3b3f33cb"
format_stamp: "Formatted at 2026-06-19 18:49:07 on dist-test-slave-m9wg"
server_key: "d63d639874f86946b431dbb707b7794f"
server_key_iv: "f1a13c2f4951cb0281d3eb2e80358dc8"
server_key_version: "kuduclusterkey@0"
I20260619 18:49:07.415401 15441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/minicluster-data/ts-0/wal/instance:
uuid: "0e48a2270b0f41a3a463e5aa3b3f33cb"
format_stamp: "Formatted at 2026-06-19 18:49:07 on dist-test-slave-m9wg"
server_key: "d63d639874f86946b431dbb707b7794f"
server_key_iv: "f1a13c2f4951cb0281d3eb2e80358dc8"
server_key_version: "kuduclusterkey@0"
I20260619 18:49:07.428476 15441 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.007s sys 0.000s
I20260619 18:49:07.455508 15461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:49:07.457491 15441 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.006s sys 0.003s
I20260619 18:49:07.457698 15441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/minicluster-data/ts-0/wal
uuid: "0e48a2270b0f41a3a463e5aa3b3f33cb"
format_stamp: "Formatted at 2026-06-19 18:49:07 on dist-test-slave-m9wg"
server_key: "d63d639874f86946b431dbb707b7794f"
server_key_iv: "f1a13c2f4951cb0281d3eb2e80358dc8"
server_key_version: "kuduclusterkey@0"
I20260619 18:49:07.458011 15441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:49:07.485801 15441 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:49:07.492342 15441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:49:07.492879 15441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:49:07.494635 15441 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:49:07.497238 15441 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:49:07.497378 15441 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:49:07.497593 15441 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:49:07.497704 15441 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:49:07.639058 15441 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.65:45305
I20260619 18:49:07.639115 15573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.65:45305 every 8 connection(s)
I20260619 18:49:07.641342 15441 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/minicluster-data/ts-0/data/info.pb
I20260619 18:49:07.649837 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 15441
I20260619 18:49:07.650059 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/minicluster-data/ts-0/wal/instance
I20260619 18:49:07.650995 4045 external_mini_cluster.cc:1428] Setting key fc1749b25ed2436c9e1bf19d2d9d5365
WARNING: no policy specified for kudu/127.3.243.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.66@KRBTEST.COM" created.
Jun 19 18:49:07 dist-test-slave-m9wg krb5kdc[14885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894947, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.65@KRBTEST.COM for kudu/127.3.243.126@KRBTEST.COM
I20260619 18:49:07.714972 15577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:49:07.655792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.65:59505 (local address 127.3.243.126:40583)
0619 18:49:07.677987 (+ 22195us) server_negotiation.cc:206] Beginning negotiation
0619 18:49:07.677994 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:49:07.683256 (+ 5262us) server_negotiation.cc:407] Connection header received
0619 18:49:07.683335 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:49:07.683340 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:49:07.683435 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:49:07.683549 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:49:07.687385 (+ 3836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:49:07.688842 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:49:07.700346 (+ 11504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:49:07.701009 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:49:07.701126 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:49:07.705724 (+ 4598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:49:07.705729 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:49:07.705733 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:49:07.705781 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:49:07.710242 (+ 4461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:49:07.711574 (+ 1332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:49:07.711581 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:49:07.711585 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:49:07.711790 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:49:07.712520 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:49:07.712525 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:49:07.712528 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:49:07.712939 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:49:07.713068 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 18:49:07.714171 (+ 1103us) server_negotiation.cc:299] Negotiation successful
0619 18:49:07.714447 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22028,"thread_start_us":22215,"threads_started":1}
I20260619 18:49:07.717869 15576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:49:07.654899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:40583 (local address 127.3.243.65:59505)
0619 18:49:07.679082 (+ 24183us) negotiation.cc:107] Waiting for socket to connect
0619 18:49:07.679147 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0619 18:49:07.682735 (+ 3588us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:49:07.685085 (+ 2350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:49:07.685098 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:49:07.685995 (+ 897us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:49:07.687177 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:49:07.687205 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:49:07.689245 (+ 2040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:49:07.689251 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:49:07.700106 (+ 10855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:49:07.700170 (+ 64us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:49:07.701591 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:49:07.701602 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:49:07.701936 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:49:07.702682 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:49:07.702714 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:49:07.705467 (+ 2753us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:49:07.710462 (+ 4995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:49:07.710471 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:49:07.710492 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:49:07.711414 (+ 922us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:49:07.711931 (+ 517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:49:07.711935 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:49:07.711939 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:49:07.712382 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:49:07.713192 (+ 810us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:49:07.713201 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:49:07.713679 (+ 478us) client_negotiation.cc:769] Sending connection context
0619 18:49:07.715540 (+ 1861us) client_negotiation.cc:240] Negotiation successful
0619 18:49:07.716197 (+ 657us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23756,"thread_start_us":28679,"threads_started":1}
I20260619 18:49:07.720321 15574 heartbeater.cc:347] Connected to a master server at 127.3.243.126:40583
I20260619 18:49:07.720697 15574 heartbeater.cc:464] Registering TS with master...
I20260619 18:49:07.721810 15574 heartbeater.cc:511] Master 127.3.243.126:40583 requested a full tablet report, sending...
Jun 19 18:49:07 dist-test-slave-m9wg krb5kdc[14885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894947, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.126@KRBTEST.COM for HTTP/127.3.243.84@KRBTEST.COM
I20260619 18:49:07.724968 15361 ts_manager.cc:194] Registered new tserver with Master: 0e48a2270b0f41a3a463e5aa3b3f33cb (127.3.243.65:45305)
I20260619 18:49:07.727392 15361 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.65@KRBTEST.COM'} at 127.3.243.65:59505
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.66.keytab.
Entry for principal kudu/127.3.243.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.66.keytab.
W20260619 18:49:07.755699 4045 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.3.243.66: real 0.101s user 0.005s sys 0.012s
I20260619 18:49:07.756645 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.243.84:38513/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.66:0
--local_ip_for_outbound_sockets=127.3.243.66
--webserver_interface=127.3.243.66
--webserver_port=0
--tserver_master_addrs=127.3.243.126:40583
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.66.keytab
--principal=kudu/127.3.243.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:39537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 18:49:08 dist-test-slave-m9wg krb5kdc[14885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894947, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.126@KRBTEST.COM for HTTP/127.3.243.84@KRBTEST.COM
W20260619 18:49:08.451723 15580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:49:08.452080 15580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:49:08.452301 15580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:49:08.461926 15580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:49:08.462100 15580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:49:08.462302 15580 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.66
I20260619 18:49:08.475066 15580 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:39537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.3.243.84:38513/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.66.keytab
--principal=kudu/127.3.243.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.243.66
--webserver_port=0
--tserver_master_addrs=127.3.243.126:40583
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.66
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:49:08.476486 15580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:49:08.477910 15580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:49:08.497404 15595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:49:08.497536 15597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:49:08.498158 15594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:49:08.499576 15580 server_base.cc:1034] running on GCE node
I20260619 18:49:08.500499 15580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:49:08.502019 15580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:49:08.503317 15580 hybrid_clock.cc:648] HybridClock initialized: now 1781894948503214 us; error 96 us; skew 500 ppm
I20260619 18:49:08.503607 15580 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:49:08 dist-test-slave-m9wg krb5kdc[14885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894948, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:49:08.507545 15580 init.cc:377] Logged in from keytab as kudu/127.3.243.66@KRBTEST.COM (short username kudu)
I20260619 18:49:08.511070 15580 webserver.cc:460] Webserver started at http://127.3.243.66:40155/ using document root <none> and password file <none>
I20260619 18:49:08.511871 15580 fs_manager.cc:359] Metadata directory not provided
I20260619 18:49:08.512048 15580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:49:08.512446 15580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 18:49:08 dist-test-slave-m9wg krb5kdc[14885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894948, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.66@KRBTEST.COM for HTTP/127.3.243.84@KRBTEST.COM
I20260619 18:49:08.532116 15580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/minicluster-data/ts-1/data/instance:
uuid: "565c231180614ee7986ccd5731aee943"
format_stamp: "Formatted at 2026-06-19 18:49:08 on dist-test-slave-m9wg"
server_key: "3c17019a4266f85213cf2399b0803d11"
server_key_iv: "cb3c6546724d9b14baf0d9ecf2794d33"
server_key_version: "kuduclusterkey@0"
I20260619 18:49:08.533514 15580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/minicluster-data/ts-1/wal/instance:
uuid: "565c231180614ee7986ccd5731aee943"
format_stamp: "Formatted at 2026-06-19 18:49:08 on dist-test-slave-m9wg"
server_key: "3c17019a4266f85213cf2399b0803d11"
server_key_iv: "cb3c6546724d9b14baf0d9ecf2794d33"
server_key_version: "kuduclusterkey@0"
I20260619 18:49:08.541540 15580 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260619 18:49:08.559872 15604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:49:08.561803 15580 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 18:49:08.562031 15580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/minicluster-data/ts-1/wal
uuid: "565c231180614ee7986ccd5731aee943"
format_stamp: "Formatted at 2026-06-19 18:49:08 on dist-test-slave-m9wg"
server_key: "3c17019a4266f85213cf2399b0803d11"
server_key_iv: "cb3c6546724d9b14baf0d9ecf2794d33"
server_key_version: "kuduclusterkey@0"
I20260619 18:49:08.562330 15580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:49:08.585880 15580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:49:08.591954 15580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:49:08.592476 15580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:49:08.594158 15580 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:49:08.596588 15580 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:49:08.596719 15580 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:49:08.596927 15580 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:49:08.597030 15580 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:49:08.692351 15580 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.66:41791
I20260619 18:49:08.692456 15716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.66:41791 every 8 connection(s)
I20260619 18:49:08.693933 15580 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/minicluster-data/ts-1/data/info.pb
I20260619 18:49:08.697084 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 15580
I20260619 18:49:08.697275 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/minicluster-data/ts-1/wal/instance
I20260619 18:49:08.697870 4045 external_mini_cluster.cc:1428] Setting key 163d2bb0684cd27839e509b39aaa173b
Jun 19 18:49:08 dist-test-slave-m9wg krb5kdc[14885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894948, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.66@KRBTEST.COM for kudu/127.3.243.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.243.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.67@KRBTEST.COM" created.
I20260619 18:49:08.731837 15574 heartbeater.cc:503] Master 127.3.243.126:40583 was elected leader, sending a full tablet report...
I20260619 18:49:08.734272 15720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:49:08.698746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:40583 (local address 127.3.243.66:42287)
0619 18:49:08.709277 (+ 10531us) negotiation.cc:107] Waiting for socket to connect
0619 18:49:08.709342 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0619 18:49:08.712472 (+ 3130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:49:08.713357 (+ 885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:49:08.713370 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:49:08.714198 (+ 828us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:49:08.715406 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:49:08.715432 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:49:08.717212 (+ 1780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:49:08.717217 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:49:08.719528 (+ 2311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:49:08.719563 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:49:08.720848 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:49:08.720856 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:49:08.721224 (+ 368us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:49:08.722062 (+ 838us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:49:08.722094 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:49:08.725643 (+ 3549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:49:08.729964 (+ 4321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:49:08.729973 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:49:08.729992 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:49:08.730400 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:49:08.730790 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:49:08.730795 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:49:08.730799 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:49:08.731000 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:49:08.731570 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:49:08.731579 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:49:08.732004 (+ 425us) client_negotiation.cc:769] Sending connection context
0619 18:49:08.732320 (+ 316us) client_negotiation.cc:240] Negotiation successful
0619 18:49:08.732865 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10100,"thread_start_us":9772,"threads_started":1}
I20260619 18:49:08.735205 15719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:49:08.697886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.66:42287 (local address 127.3.243.126:40583)
0619 18:49:08.711399 (+ 13513us) server_negotiation.cc:206] Beginning negotiation
0619 18:49:08.711406 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:49:08.711429 (+ 23us) server_negotiation.cc:407] Connection header received
0619 18:49:08.712806 (+ 1377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:49:08.712813 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:49:08.712935 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:49:08.713057 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:49:08.715581 (+ 2524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:49:08.717040 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:49:08.719707 (+ 2667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:49:08.720551 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:49:08.720708 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:49:08.725937 (+ 5229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:49:08.725945 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:49:08.725950 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:49:08.726009 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:49:08.729810 (+ 3801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:49:08.730513 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:49:08.730517 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:49:08.730520 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:49:08.730678 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:49:08.731106 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:49:08.731110 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:49:08.731112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:49:08.731430 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:49:08.734080 (+ 2650us) server_negotiation.cc:1035] Waiting for connection context
0619 18:49:08.734243 (+ 163us) server_negotiation.cc:299] Negotiation successful
0619 18:49:08.734728 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13387,"thread_start_us":13251,"threads_started":1}
I20260619 18:49:08.736474 15717 heartbeater.cc:347] Connected to a master server at 127.3.243.126:40583
I20260619 18:49:08.736805 15717 heartbeater.cc:464] Registering TS with master...
I20260619 18:49:08.737841 15717 heartbeater.cc:511] Master 127.3.243.126:40583 requested a full tablet report, sending...
I20260619 18:49:08.739733 15361 ts_manager.cc:194] Registered new tserver with Master: 565c231180614ee7986ccd5731aee943 (127.3.243.66:41791)
I20260619 18:49:08.741000 15361 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.66@KRBTEST.COM'} at 127.3.243.66:42287
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.67.keytab.
Entry for principal kudu/127.3.243.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.67.keytab.
I20260619 18:49:08.757223 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.243.84:38513/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.67:0
--local_ip_for_outbound_sockets=127.3.243.67
--webserver_interface=127.3.243.67
--webserver_port=0
--tserver_master_addrs=127.3.243.126:40583
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.67.keytab
--principal=kudu/127.3.243.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:39537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:49:09.357877 15723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:49:09.358253 15723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:49:09.358359 15723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:49:09.367695 15723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:49:09.367810 15723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:49:09.367928 15723 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.67
I20260619 18:49:09.381083 15723 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:39537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.3.243.84:38513/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.67.keytab
--principal=kudu/127.3.243.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.243.67
--webserver_port=0
--tserver_master_addrs=127.3.243.126:40583
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.67
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:49:09.382396 15723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:49:09.383746 15723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:49:09.400933 15728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:49:09.400991 15731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:49:09.401042 15729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:49:09.402520 15723 server_base.cc:1034] running on GCE node
I20260619 18:49:09.403790 15723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:49:09.405273 15723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:49:09.406580 15723 hybrid_clock.cc:648] HybridClock initialized: now 1781894949406539 us; error 61 us; skew 500 ppm
I20260619 18:49:09.406910 15723 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:49:09 dist-test-slave-m9wg krb5kdc[14885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894949, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:49:09.412890 15723 init.cc:377] Logged in from keytab as kudu/127.3.243.67@KRBTEST.COM (short username kudu)
I20260619 18:49:09.414691 15723 webserver.cc:460] Webserver started at http://127.3.243.67:37159/ using document root <none> and password file <none>
I20260619 18:49:09.415810 15723 fs_manager.cc:359] Metadata directory not provided
I20260619 18:49:09.415930 15723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:49:09.416524 15723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 18:49:09 dist-test-slave-m9wg krb5kdc[14885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894949, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.67@KRBTEST.COM for HTTP/127.3.243.84@KRBTEST.COM
I20260619 18:49:09.438963 15723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/minicluster-data/ts-2/data/instance:
uuid: "f09ff6644f2146e5a4ec517407ff6326"
format_stamp: "Formatted at 2026-06-19 18:49:09 on dist-test-slave-m9wg"
server_key: "a69cad1d4bebba64c2caa2e16c4f36f7"
server_key_iv: "6fbae251ddad94866ddba6d5243a3905"
server_key_version: "kuduclusterkey@0"
I20260619 18:49:09.439980 15723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/minicluster-data/ts-2/wal/instance:
uuid: "f09ff6644f2146e5a4ec517407ff6326"
format_stamp: "Formatted at 2026-06-19 18:49:09 on dist-test-slave-m9wg"
server_key: "a69cad1d4bebba64c2caa2e16c4f36f7"
server_key_iv: "6fbae251ddad94866ddba6d5243a3905"
server_key_version: "kuduclusterkey@0"
I20260619 18:49:09.451278 15723 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.008s sys 0.000s
I20260619 18:49:09.476382 15738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:49:09.478415 15723 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260619 18:49:09.478634 15723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/minicluster-data/ts-2/wal
uuid: "f09ff6644f2146e5a4ec517407ff6326"
format_stamp: "Formatted at 2026-06-19 18:49:09 on dist-test-slave-m9wg"
server_key: "a69cad1d4bebba64c2caa2e16c4f36f7"
server_key_iv: "6fbae251ddad94866ddba6d5243a3905"
server_key_version: "kuduclusterkey@0"
I20260619 18:49:09.478921 15723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:49:09.530655 15723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:49:09.540314 15723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:49:09.541124 15723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:49:09.542948 15723 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:49:09.547426 15723 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:49:09.547734 15723 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:49:09.547986 15723 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:49:09.548084 15723 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:49:09.653471 15723 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.67:33355
I20260619 18:49:09.653738 15850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.67:33355 every 8 connection(s)
I20260619 18:49:09.655754 15723 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/minicluster-data/ts-2/data/info.pb
I20260619 18:49:09.657972 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 15723
I20260619 18:49:09.658450 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/minicluster-data/ts-2/wal/instance
I20260619 18:49:09.659287 4045 external_mini_cluster.cc:1428] Setting key 8cb6873761c1904ee8e088cb46651cdd
Jun 19 18:49:09 dist-test-slave-m9wg krb5kdc[14885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894949, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.67@KRBTEST.COM for kudu/127.3.243.126@KRBTEST.COM
I20260619 18:49:09.699409 15853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:49:09.661473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.67:48911 (local address 127.3.243.126:40583)
0619 18:49:09.668973 (+ 7500us) server_negotiation.cc:206] Beginning negotiation
0619 18:49:09.668982 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:49:09.669004 (+ 22us) server_negotiation.cc:407] Connection header received
0619 18:49:09.671516 (+ 2512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:49:09.671520 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:49:09.671597 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:49:09.671688 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:49:09.674790 (+ 3102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:49:09.676125 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:49:09.679978 (+ 3853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:49:09.681229 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:49:09.681430 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:49:09.687545 (+ 6115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:49:09.687551 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:49:09.687555 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:49:09.687608 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:49:09.693127 (+ 5519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:49:09.694275 (+ 1148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:49:09.694280 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:49:09.694283 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:49:09.694473 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:49:09.695350 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:49:09.695354 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:49:09.695357 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:49:09.696009 (+ 652us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:49:09.696306 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0619 18:49:09.698100 (+ 1794us) server_negotiation.cc:299] Negotiation successful
0619 18:49:09.698634 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7164,"thread_start_us":6547,"threads_started":1}
I20260619 18:49:09.701213 15854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:49:09.662963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:40583 (local address 127.3.243.67:48911)
0619 18:49:09.666325 (+ 3362us) negotiation.cc:107] Waiting for socket to connect
0619 18:49:09.666721 (+ 396us) client_negotiation.cc:174] Beginning negotiation
0619 18:49:09.671194 (+ 4473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:49:09.671953 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:49:09.671964 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:49:09.673269 (+ 1305us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:49:09.674622 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:49:09.674643 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:49:09.676449 (+ 1806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:49:09.676479 (+ 30us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:49:09.679696 (+ 3217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:49:09.679741 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:49:09.681718 (+ 1977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:49:09.681728 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:49:09.682171 (+ 443us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:49:09.683129 (+ 958us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:49:09.683158 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:49:09.687194 (+ 4036us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:49:09.693310 (+ 6116us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:49:09.693320 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:49:09.693347 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:49:09.694107 (+ 760us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:49:09.694590 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:49:09.694595 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:49:09.694609 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:49:09.695219 (+ 610us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:49:09.696256 (+ 1037us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:49:09.696304 (+ 48us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:49:09.697361 (+ 1057us) client_negotiation.cc:769] Sending connection context
0619 18:49:09.698012 (+ 651us) client_negotiation.cc:240] Negotiation successful
0619 18:49:09.698988 (+ 976us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2162,"thread_start_us":1325,"threads_started":1}
I20260619 18:49:09.703488 15851 heartbeater.cc:347] Connected to a master server at 127.3.243.126:40583
I20260619 18:49:09.704008 15851 heartbeater.cc:464] Registering TS with master...
I20260619 18:49:09.705250 15851 heartbeater.cc:511] Master 127.3.243.126:40583 requested a full tablet report, sending...
I20260619 18:49:09.707523 15361 ts_manager.cc:194] Registered new tserver with Master: f09ff6644f2146e5a4ec517407ff6326 (127.3.243.67:33355)
I20260619 18:49:09.709298 4045 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 18:49:09.709287 15361 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.67@KRBTEST.COM'} at 127.3.243.67:48911
I20260619 18:49:09.745193 15853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:49:09.720400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45704 (local address 127.3.243.126:40583)
0619 18:49:09.720623 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0619 18:49:09.720627 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:49:09.722040 (+ 1413us) server_negotiation.cc:407] Connection header received
0619 18:49:09.722462 (+ 422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:49:09.722466 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:49:09.722606 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:49:09.722694 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:49:09.725470 (+ 2776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:49:09.727246 (+ 1776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:49:09.730047 (+ 2801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:49:09.731120 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:49:09.731357 (+ 237us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:49:09.735388 (+ 4031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:49:09.735393 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:49:09.735397 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:49:09.735432 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:49:09.740458 (+ 5026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:49:09.741156 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:49:09.741160 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:49:09.741162 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:49:09.741321 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:49:09.742083 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:49:09.742087 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:49:09.742089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:49:09.743211 (+ 1122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:49:09.743632 (+ 421us) server_negotiation.cc:1035] Waiting for connection context
0619 18:49:09.744279 (+ 647us) server_negotiation.cc:299] Negotiation successful
0619 18:49:09.744699 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20260619 18:49:09.745141 15717 heartbeater.cc:503] Master 127.3.243.126:40583 was elected leader, sending a full tablet report...
I20260619 18:49:09.752206 15360 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45704:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:49:09.767722 15360 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:49:09.829461 15868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:49:09.792770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.67:33355 (local address 127.0.0.1:37104)
0619 18:49:09.809773 (+ 17003us) negotiation.cc:107] Waiting for socket to connect
0619 18:49:09.809796 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 18:49:09.809933 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:49:09.811348 (+ 1415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:49:09.811356 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:49:09.811385 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:49:09.813894 (+ 2509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:49:09.813920 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:49:09.818030 (+ 4110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:49:09.818035 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:49:09.822949 (+ 4914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:49:09.822964 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:49:09.827010 (+ 4046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:49:09.827018 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:49:09.827409 (+ 391us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:49:09.827533 (+ 124us) client_negotiation.cc:769] Sending connection context
0619 18:49:09.827969 (+ 436us) client_negotiation.cc:240] Negotiation successful
0619 18:49:09.828324 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16797,"mutex_wait_us":5,"thread_start_us":16165,"threads_started":1}
I20260619 18:49:09.830322 15869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:49:09.793550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37104 (local address 127.3.243.67:33355)
0619 18:49:09.810336 (+ 16786us) server_negotiation.cc:206] Beginning negotiation
0619 18:49:09.810343 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:49:09.810398 (+ 55us) server_negotiation.cc:407] Connection header received
0619 18:49:09.810508 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:49:09.810514 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:49:09.810904 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:49:09.811195 (+ 291us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:49:09.815003 (+ 3808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:49:09.817803 (+ 2800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:49:09.823157 (+ 5354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:49:09.826379 (+ 3222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:49:09.826562 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:49:09.827419 (+ 857us) server_negotiation.cc:1035] Waiting for connection context
0619 18:49:09.829409 (+ 1990us) server_negotiation.cc:299] Negotiation successful
0619 18:49:09.829720 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16459,"thread_start_us":13916,"threads_started":1}
I20260619 18:49:09.830891 15864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:49:09.790124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.66:41791 (local address 127.0.0.1:51834)
0619 18:49:09.792650 (+ 2526us) negotiation.cc:107] Waiting for socket to connect
0619 18:49:09.792760 (+ 110us) client_negotiation.cc:174] Beginning negotiation
0619 18:49:09.793008 (+ 248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:49:09.811964 (+ 18956us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:49:09.811969 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:49:09.811995 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:49:09.814753 (+ 2758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:49:09.814766 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:49:09.819413 (+ 4647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:49:09.819418 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:49:09.823746 (+ 4328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:49:09.823819 (+ 73us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:49:09.829432 (+ 5613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:49:09.829440 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:49:09.829849 (+ 409us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:49:09.829883 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 18:49:09.829965 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 18:49:09.830100 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1982,"mutex_wait_us":23,"thread_start_us":1453,"threads_started":1}
I20260619 18:49:09.831864 15867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:49:09.791710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37402 (local address 127.3.243.65:45305)
0619 18:49:09.810353 (+ 18643us) server_negotiation.cc:206] Beginning negotiation
0619 18:49:09.810361 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:49:09.810389 (+ 28us) server_negotiation.cc:407] Connection header received
0619 18:49:09.810493 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:49:09.810501 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:49:09.810919 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:49:09.811407 (+ 488us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:49:09.815027 (+ 3620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:49:09.818954 (+ 3927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:49:09.824588 (+ 5634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:49:09.828977 (+ 4389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:49:09.829102 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:49:09.829477 (+ 375us) server_negotiation.cc:1035] Waiting for connection context
0619 18:49:09.831100 (+ 1623us) server_negotiation.cc:299] Negotiation successful
0619 18:49:09.831298 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18422,"thread_start_us":18163,"threads_started":1}
I20260619 18:49:09.832333 15866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:49:09.790770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51834 (local address 127.3.243.66:41791)
0619 18:49:09.809911 (+ 19141us) server_negotiation.cc:206] Beginning negotiation
0619 18:49:09.809922 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 18:49:09.809965 (+ 43us) server_negotiation.cc:407] Connection header received
0619 18:49:09.810839 (+ 874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:49:09.810849 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:49:09.811359 (+ 510us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:49:09.812386 (+ 1027us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:49:09.815277 (+ 2891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:49:09.819026 (+ 3749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:49:09.823944 (+ 4918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:49:09.828927 (+ 4983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:49:09.831198 (+ 2271us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:49:09.831455 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0619 18:49:09.831520 (+ 65us) server_negotiation.cc:299] Negotiation successful
0619 18:49:09.831662 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18760,"thread_start_us":18008,"threads_started":1}
I20260619 18:49:09.832865 15865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:49:09.790349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.65:45305 (local address 127.0.0.1:37402)
0619 18:49:09.805347 (+ 14998us) negotiation.cc:107] Waiting for socket to connect
0619 18:49:09.805515 (+ 168us) client_negotiation.cc:174] Beginning negotiation
0619 18:49:09.805731 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:49:09.812084 (+ 6353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:49:09.812088 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:49:09.812254 (+ 166us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:49:09.814565 (+ 2311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:49:09.814663 (+ 98us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:49:09.819228 (+ 4565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:49:09.819236 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:49:09.824295 (+ 5059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:49:09.824309 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:49:09.829847 (+ 5538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:49:09.829855 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:49:09.830089 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:49:09.830121 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 18:49:09.830196 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 18:49:09.830341 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14529,"mutex_wait_us":84,"thread_start_us":15576,"threads_started":1}
I20260619 18:49:09.838299 15509 tablet_service.cc:1467] Processing CreateTablet for tablet b91a6f3a72fa4f56b62b216afbad693b (DEFAULT_TABLE table=test-table [id=e8cef817dee141f69de9f393587dfd75]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:49:09.838027 15786 tablet_service.cc:1467] Processing CreateTablet for tablet b91a6f3a72fa4f56b62b216afbad693b (DEFAULT_TABLE table=test-table [id=e8cef817dee141f69de9f393587dfd75]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:49:09.839447 15652 tablet_service.cc:1467] Processing CreateTablet for tablet b91a6f3a72fa4f56b62b216afbad693b (DEFAULT_TABLE table=test-table [id=e8cef817dee141f69de9f393587dfd75]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:49:09.841261 15509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b91a6f3a72fa4f56b62b216afbad693b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:49:09.841310 15786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b91a6f3a72fa4f56b62b216afbad693b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:49:09.842399 15652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b91a6f3a72fa4f56b62b216afbad693b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:49:09.864372 15870 tablet_bootstrap.cc:492] T b91a6f3a72fa4f56b62b216afbad693b P 0e48a2270b0f41a3a463e5aa3b3f33cb: Bootstrap starting.
I20260619 18:49:09.871981 15871 tablet_bootstrap.cc:492] T b91a6f3a72fa4f56b62b216afbad693b P f09ff6644f2146e5a4ec517407ff6326: Bootstrap starting.
I20260619 18:49:09.875392 15870 tablet_bootstrap.cc:654] T b91a6f3a72fa4f56b62b216afbad693b P 0e48a2270b0f41a3a463e5aa3b3f33cb: Neither blocks nor log segments found. Creating new log.
I20260619 18:49:09.878983 15870 log.cc:824] T b91a6f3a72fa4f56b62b216afbad693b P 0e48a2270b0f41a3a463e5aa3b3f33cb: Log is configured to *not* fsync() on all Append() calls
I20260619 18:49:09.879576 15871 tablet_bootstrap.cc:654] T b91a6f3a72fa4f56b62b216afbad693b P f09ff6644f2146e5a4ec517407ff6326: Neither blocks nor log segments found. Creating new log.
I20260619 18:49:09.883832 15871 log.cc:824] T b91a6f3a72fa4f56b62b216afbad693b P f09ff6644f2146e5a4ec517407ff6326: Log is configured to *not* fsync() on all Append() calls
I20260619 18:49:09.894434 15870 tablet_bootstrap.cc:492] T b91a6f3a72fa4f56b62b216afbad693b P 0e48a2270b0f41a3a463e5aa3b3f33cb: No bootstrap required, opened a new log
I20260619 18:49:09.895236 15870 ts_tablet_manager.cc:1397] T b91a6f3a72fa4f56b62b216afbad693b P 0e48a2270b0f41a3a463e5aa3b3f33cb: Time spent bootstrapping tablet: real 0.031s user 0.012s sys 0.013s
I20260619 18:49:09.896692 15871 tablet_bootstrap.cc:492] T b91a6f3a72fa4f56b62b216afbad693b P f09ff6644f2146e5a4ec517407ff6326: No bootstrap required, opened a new log
I20260619 18:49:09.897377 15871 ts_tablet_manager.cc:1397] T b91a6f3a72fa4f56b62b216afbad693b P f09ff6644f2146e5a4ec517407ff6326: Time spent bootstrapping tablet: real 0.026s user 0.015s sys 0.006s
I20260619 18:49:09.897929 15872 tablet_bootstrap.cc:492] T b91a6f3a72fa4f56b62b216afbad693b P 565c231180614ee7986ccd5731aee943: Bootstrap starting.
I20260619 18:49:09.906850 15872 tablet_bootstrap.cc:654] T b91a6f3a72fa4f56b62b216afbad693b P 565c231180614ee7986ccd5731aee943: Neither blocks nor log segments found. Creating new log.
I20260619 18:49:09.911358 15872 log.cc:824] T b91a6f3a72fa4f56b62b216afbad693b P 565c231180614ee7986ccd5731aee943: Log is configured to *not* fsync() on all Append() calls
I20260619 18:49:09.922642 15872 tablet_bootstrap.cc:492] T b91a6f3a72fa4f56b62b216afbad693b P 565c231180614ee7986ccd5731aee943: No bootstrap required, opened a new log
I20260619 18:49:09.922118 15871 raft_consensus.cc:348] T b91a6f3a72fa4f56b62b216afbad693b P f09ff6644f2146e5a4ec517407ff6326 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565c231180614ee7986ccd5731aee943" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 41791 } } peers { permanent_uuid: "0e48a2270b0f41a3a463e5aa3b3f33cb" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 45305 } } peers { permanent_uuid: "f09ff6644f2146e5a4ec517407ff6326" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 33355 } }
I20260619 18:49:09.923197 15871 raft_consensus.cc:374] T b91a6f3a72fa4f56b62b216afbad693b P f09ff6644f2146e5a4ec517407ff6326 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:49:09.923585 15871 raft_consensus.cc:729] T b91a6f3a72fa4f56b62b216afbad693b P f09ff6644f2146e5a4ec517407ff6326 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f09ff6644f2146e5a4ec517407ff6326, State: Initialized, Role: FOLLOWER
I20260619 18:49:09.923686 15872 ts_tablet_manager.cc:1397] T b91a6f3a72fa4f56b62b216afbad693b P 565c231180614ee7986ccd5731aee943: Time spent bootstrapping tablet: real 0.026s user 0.007s sys 0.017s
I20260619 18:49:09.925244 15870 raft_consensus.cc:348] T b91a6f3a72fa4f56b62b216afbad693b P 0e48a2270b0f41a3a463e5aa3b3f33cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565c231180614ee7986ccd5731aee943" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 41791 } } peers { permanent_uuid: "0e48a2270b0f41a3a463e5aa3b3f33cb" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 45305 } } peers { permanent_uuid: "f09ff6644f2146e5a4ec517407ff6326" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 33355 } }
I20260619 18:49:09.926470 15870 raft_consensus.cc:374] T b91a6f3a72fa4f56b62b216afbad693b P 0e48a2270b0f41a3a463e5aa3b3f33cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:49:09.926626 15870 raft_consensus.cc:729] T b91a6f3a72fa4f56b62b216afbad693b P 0e48a2270b0f41a3a463e5aa3b3f33cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e48a2270b0f41a3a463e5aa3b3f33cb, State: Initialized, Role: FOLLOWER
I20260619 18:49:09.927518 15870 consensus_queue.cc:260] T b91a6f3a72fa4f56b62b216afbad693b P 0e48a2270b0f41a3a463e5aa3b3f33cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565c231180614ee7986ccd5731aee943" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 41791 } } peers { permanent_uuid: "0e48a2270b0f41a3a463e5aa3b3f33cb" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 45305 } } peers { permanent_uuid: "f09ff6644f2146e5a4ec517407ff6326" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 33355 } }
I20260619 18:49:09.925838 15871 consensus_queue.cc:260] T b91a6f3a72fa4f56b62b216afbad693b P f09ff6644f2146e5a4ec517407ff6326 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565c231180614ee7986ccd5731aee943" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 41791 } } peers { permanent_uuid: "0e48a2270b0f41a3a463e5aa3b3f33cb" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 45305 } } peers { permanent_uuid: "f09ff6644f2146e5a4ec517407ff6326" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 33355 } }
I20260619 18:49:09.930701 15870 ts_tablet_manager.cc:1428] T b91a6f3a72fa4f56b62b216afbad693b P 0e48a2270b0f41a3a463e5aa3b3f33cb: Time spent starting tablet: real 0.035s user 0.026s sys 0.003s
I20260619 18:49:09.931779 15851 heartbeater.cc:503] Master 127.3.243.126:40583 was elected leader, sending a full tablet report...
I20260619 18:49:09.932720 15871 ts_tablet_manager.cc:1428] T b91a6f3a72fa4f56b62b216afbad693b P f09ff6644f2146e5a4ec517407ff6326: Time spent starting tablet: real 0.035s user 0.025s sys 0.003s
I20260619 18:49:09.947566 15876 raft_consensus.cc:482] T b91a6f3a72fa4f56b62b216afbad693b P 0e48a2270b0f41a3a463e5aa3b3f33cb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:49:09.948222 15876 raft_consensus.cc:504] T b91a6f3a72fa4f56b62b216afbad693b P 0e48a2270b0f41a3a463e5aa3b3f33cb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565c231180614ee7986ccd5731aee943" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 41791 } } peers { permanent_uuid: "0e48a2270b0f41a3a463e5aa3b3f33cb" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 45305 } } peers { permanent_uuid: "f09ff6644f2146e5a4ec517407ff6326" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 33355 } }
I20260619 18:49:09.952973 15876 leader_election.cc:288] T b91a6f3a72fa4f56b62b216afbad693b P 0e48a2270b0f41a3a463e5aa3b3f33cb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 565c231180614ee7986ccd5731aee943 (127.3.243.66:41791), f09ff6644f2146e5a4ec517407ff6326 (127.3.243.67:33355)
I20260619 18:49:09.953760 15872 raft_consensus.cc:348] T b91a6f3a72fa4f56b62b216afbad693b P 565c231180614ee7986ccd5731aee943 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565c231180614ee7986ccd5731aee943" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 41791 } } peers { permanent_uuid: "0e48a2270b0f41a3a463e5aa3b3f33cb" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 45305 } } peers { permanent_uuid: "f09ff6644f2146e5a4ec517407ff6326" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 33355 } }
I20260619 18:49:09.954514 15872 raft_consensus.cc:374] T b91a6f3a72fa4f56b62b216afbad693b P 565c231180614ee7986ccd5731aee943 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:49:09.954813 15872 raft_consensus.cc:729] T b91a6f3a72fa4f56b62b216afbad693b P 565c231180614ee7986ccd5731aee943 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 565c231180614ee7986ccd5731aee943, State: Initialized, Role: FOLLOWER
I20260619 18:49:09.956203 15872 consensus_queue.cc:260] T b91a6f3a72fa4f56b62b216afbad693b P 565c231180614ee7986ccd5731aee943 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565c231180614ee7986ccd5731aee943" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 41791 } } peers { permanent_uuid: "0e48a2270b0f41a3a463e5aa3b3f33cb" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 45305 } } peers { permanent_uuid: "f09ff6644f2146e5a4ec517407ff6326" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 33355 } }
I20260619 18:49:09.959437 15872 ts_tablet_manager.cc:1428] T b91a6f3a72fa4f56b62b216afbad693b P 565c231180614ee7986ccd5731aee943: Time spent starting tablet: real 0.035s user 0.021s sys 0.005s
I20260619 18:49:09.967819 15869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:49:09.953813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.65:33719 (local address 127.3.243.67:33355)
0619 18:49:09.954030 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0619 18:49:09.954037 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:49:09.955605 (+ 1568us) server_negotiation.cc:407] Connection header received
0619 18:49:09.955906 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:49:09.955909 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:49:09.955973 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:49:09.956210 (+ 237us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:49:09.957034 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:49:09.958407 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:49:09.963628 (+ 5221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:49:09.966346 (+ 2718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:49:09.966549 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:49:09.966616 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 18:49:09.967171 (+ 555us) server_negotiation.cc:299] Negotiation successful
0619 18:49:09.967359 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 18:49:09.967953 15866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:49:09.953903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.65:36519 (local address 127.3.243.66:41791)
0619 18:49:09.954111 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0619 18:49:09.954117 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:49:09.955635 (+ 1518us) server_negotiation.cc:407] Connection header received
0619 18:49:09.956226 (+ 591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:49:09.956232 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:49:09.956580 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:49:09.956658 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:49:09.958150 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:49:09.960095 (+ 1945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:49:09.964344 (+ 4249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:49:09.966402 (+ 2058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:49:09.966533 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:49:09.966603 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 18:49:09.967339 (+ 736us) server_negotiation.cc:299] Negotiation successful
0619 18:49:09.967489 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 18:49:09.968058 15879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:49:09.953368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.66:41791 (local address 127.3.243.65:36519)
0619 18:49:09.955537 (+ 2169us) negotiation.cc:107] Waiting for socket to connect
0619 18:49:09.955555 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 18:49:09.955882 (+ 327us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:49:09.956998 (+ 1116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:49:09.957002 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:49:09.957027 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:49:09.957992 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:49:09.958005 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:49:09.960724 (+ 2719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:49:09.960729 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:49:09.964087 (+ 3358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:49:09.964103 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:49:09.966899 (+ 2796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:49:09.966905 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:49:09.967082 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:49:09.967189 (+ 107us) client_negotiation.cc:769] Sending connection context
0619 18:49:09.967279 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 18:49:09.967415 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1881,"mutex_wait_us":39,"thread_start_us":1734,"threads_started":1}
I20260619 18:49:09.968385 15878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:49:09.953518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.67:33355 (local address 127.3.243.65:33719)
0619 18:49:09.955503 (+ 1985us) negotiation.cc:107] Waiting for socket to connect
0619 18:49:09.955521 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 18:49:09.955689 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:49:09.956220 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:49:09.956224 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:49:09.956246 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:49:09.956874 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:49:09.956889 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:49:09.958590 (+ 1701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:49:09.958595 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:49:09.963265 (+ 4670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:49:09.963280 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:49:09.966614 (+ 3334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:49:09.966620 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:49:09.966902 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:49:09.966966 (+ 64us) client_negotiation.cc:769] Sending connection context
0619 18:49:09.967043 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 18:49:09.967216 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1819,"mutex_wait_us":26,"thread_start_us":1089,"threads_started":1}
I20260619 18:49:09.970357 15672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b91a6f3a72fa4f56b62b216afbad693b" candidate_uuid: "0e48a2270b0f41a3a463e5aa3b3f33cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "565c231180614ee7986ccd5731aee943" is_pre_election: true
I20260619 18:49:09.970818 15806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b91a6f3a72fa4f56b62b216afbad693b" candidate_uuid: "0e48a2270b0f41a3a463e5aa3b3f33cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f09ff6644f2146e5a4ec517407ff6326" is_pre_election: true
I20260619 18:49:09.971347 15806 raft_consensus.cc:2454] T b91a6f3a72fa4f56b62b216afbad693b P f09ff6644f2146e5a4ec517407ff6326 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e48a2270b0f41a3a463e5aa3b3f33cb in term 0.
I20260619 18:49:09.971441 15672 raft_consensus.cc:2454] T b91a6f3a72fa4f56b62b216afbad693b P 565c231180614ee7986ccd5731aee943 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e48a2270b0f41a3a463e5aa3b3f33cb in term 0.
I20260619 18:49:09.972350 15462 leader_election.cc:302] T b91a6f3a72fa4f56b62b216afbad693b P 0e48a2270b0f41a3a463e5aa3b3f33cb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e48a2270b0f41a3a463e5aa3b3f33cb, f09ff6644f2146e5a4ec517407ff6326; no voters:
I20260619 18:49:09.973106 15876 raft_consensus.cc:2781] T b91a6f3a72fa4f56b62b216afbad693b P 0e48a2270b0f41a3a463e5aa3b3f33cb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:49:09.973445 15876 raft_consensus.cc:482] T b91a6f3a72fa4f56b62b216afbad693b P 0e48a2270b0f41a3a463e5aa3b3f33cb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:49:09.973685 15876 raft_consensus.cc:3037] T b91a6f3a72fa4f56b62b216afbad693b P 0e48a2270b0f41a3a463e5aa3b3f33cb [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:49:09.979856 15876 raft_consensus.cc:504] T b91a6f3a72fa4f56b62b216afbad693b P 0e48a2270b0f41a3a463e5aa3b3f33cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565c231180614ee7986ccd5731aee943" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 41791 } } peers { permanent_uuid: "0e48a2270b0f41a3a463e5aa3b3f33cb" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 45305 } } peers { permanent_uuid: "f09ff6644f2146e5a4ec517407ff6326" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 33355 } }
I20260619 18:49:09.981464 15876 leader_election.cc:288] T b91a6f3a72fa4f56b62b216afbad693b P 0e48a2270b0f41a3a463e5aa3b3f33cb [CANDIDATE]: Term 1 election: Requested vote from peers 565c231180614ee7986ccd5731aee943 (127.3.243.66:41791), f09ff6644f2146e5a4ec517407ff6326 (127.3.243.67:33355)
I20260619 18:49:09.982486 15672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b91a6f3a72fa4f56b62b216afbad693b" candidate_uuid: "0e48a2270b0f41a3a463e5aa3b3f33cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "565c231180614ee7986ccd5731aee943"
I20260619 18:49:09.982897 15806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b91a6f3a72fa4f56b62b216afbad693b" candidate_uuid: "0e48a2270b0f41a3a463e5aa3b3f33cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f09ff6644f2146e5a4ec517407ff6326"
I20260619 18:49:09.983112 15672 raft_consensus.cc:3037] T b91a6f3a72fa4f56b62b216afbad693b P 565c231180614ee7986ccd5731aee943 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:49:09.983355 15806 raft_consensus.cc:3037] T b91a6f3a72fa4f56b62b216afbad693b P f09ff6644f2146e5a4ec517407ff6326 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:49:09.992779 15806 raft_consensus.cc:2454] T b91a6f3a72fa4f56b62b216afbad693b P f09ff6644f2146e5a4ec517407ff6326 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e48a2270b0f41a3a463e5aa3b3f33cb in term 1.
I20260619 18:49:09.992816 15672 raft_consensus.cc:2454] T b91a6f3a72fa4f56b62b216afbad693b P 565c231180614ee7986ccd5731aee943 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e48a2270b0f41a3a463e5aa3b3f33cb in term 1.
I20260619 18:49:09.993580 15462 leader_election.cc:302] T b91a6f3a72fa4f56b62b216afbad693b P 0e48a2270b0f41a3a463e5aa3b3f33cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0e48a2270b0f41a3a463e5aa3b3f33cb, 565c231180614ee7986ccd5731aee943, f09ff6644f2146e5a4ec517407ff6326; no voters:
I20260619 18:49:09.993974 15876 raft_consensus.cc:2781] T b91a6f3a72fa4f56b62b216afbad693b P 0e48a2270b0f41a3a463e5aa3b3f33cb [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:49:09.994679 15876 raft_consensus.cc:686] T b91a6f3a72fa4f56b62b216afbad693b P 0e48a2270b0f41a3a463e5aa3b3f33cb [term 1 LEADER]: Becoming Leader. State: Replica: 0e48a2270b0f41a3a463e5aa3b3f33cb, State: Running, Role: LEADER
I20260619 18:49:09.995414 15876 consensus_queue.cc:237] T b91a6f3a72fa4f56b62b216afbad693b P 0e48a2270b0f41a3a463e5aa3b3f33cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565c231180614ee7986ccd5731aee943" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 41791 } } peers { permanent_uuid: "0e48a2270b0f41a3a463e5aa3b3f33cb" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 45305 } } peers { permanent_uuid: "f09ff6644f2146e5a4ec517407ff6326" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 33355 } }
I20260619 18:49:10.004558 15361 catalog_manager.cc:5495] T b91a6f3a72fa4f56b62b216afbad693b P 0e48a2270b0f41a3a463e5aa3b3f33cb reported cstate change: term changed from 0 to 1, leader changed from <none> to 0e48a2270b0f41a3a463e5aa3b3f33cb (127.3.243.65). New cstate: current_term: 1 leader_uuid: "0e48a2270b0f41a3a463e5aa3b3f33cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565c231180614ee7986ccd5731aee943" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 41791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e48a2270b0f41a3a463e5aa3b3f33cb" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 45305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f09ff6644f2146e5a4ec517407ff6326" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 33355 } health_report { overall_health: UNKNOWN } } }
I20260619 18:49:10.038308 15867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:49:10.028300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37418 (local address 127.3.243.65:45305)
0619 18:49:10.028760 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0619 18:49:10.028766 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:49:10.028789 (+ 23us) server_negotiation.cc:407] Connection header received
0619 18:49:10.028888 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:49:10.028891 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:49:10.028991 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:49:10.029123 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:49:10.029616 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:49:10.031140 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:49:10.034585 (+ 3445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:49:10.035781 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:49:10.035918 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:49:10.036932 (+ 1014us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:49:10.037570 (+ 638us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:49:10.037718 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 18:49:10.037845 (+ 127us) server_negotiation.cc:299] Negotiation successful
0619 18:49:10.037946 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212}
I20260619 18:49:10.055786 15806 raft_consensus.cc:1261] T b91a6f3a72fa4f56b62b216afbad693b P f09ff6644f2146e5a4ec517407ff6326 [term 1 FOLLOWER]: Refusing update from remote peer 0e48a2270b0f41a3a463e5aa3b3f33cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:49:10.055773 15672 raft_consensus.cc:1261] T b91a6f3a72fa4f56b62b216afbad693b P 565c231180614ee7986ccd5731aee943 [term 1 FOLLOWER]: Refusing update from remote peer 0e48a2270b0f41a3a463e5aa3b3f33cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:49:10.057934 15876 consensus_queue.cc:1035] T b91a6f3a72fa4f56b62b216afbad693b P 0e48a2270b0f41a3a463e5aa3b3f33cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f09ff6644f2146e5a4ec517407ff6326" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 33355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:49:10.058657 15881 consensus_queue.cc:1035] T b91a6f3a72fa4f56b62b216afbad693b P 0e48a2270b0f41a3a463e5aa3b3f33cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "565c231180614ee7986ccd5731aee943" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 41791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:49:10.113999 15360 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45704:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:49:10.114873 15360 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45704:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:49:10.125003 15360 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cdd11781508e44d5a36669725919104d: Sending DeleteTablet for 3 replicas of tablet b91a6f3a72fa4f56b62b216afbad693b
I20260619 18:49:10.127391 15652 tablet_service.cc:1514] Processing DeleteTablet for tablet b91a6f3a72fa4f56b62b216afbad693b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:49:10 UTC) from {username='kudu', principal='kudu/127.3.243.126@KRBTEST.COM'} at 127.0.0.1:51834
I20260619 18:49:10.127985 15786 tablet_service.cc:1514] Processing DeleteTablet for tablet b91a6f3a72fa4f56b62b216afbad693b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:49:10 UTC) from {username='kudu', principal='kudu/127.3.243.126@KRBTEST.COM'} at 127.0.0.1:37104
I20260619 18:49:10.128070 15509 tablet_service.cc:1514] Processing DeleteTablet for tablet b91a6f3a72fa4f56b62b216afbad693b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:49:10 UTC) from {username='kudu', principal='kudu/127.3.243.126@KRBTEST.COM'} at 127.0.0.1:37402
I20260619 18:49:10.134538 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 15441
I20260619 18:49:10.136777 15895 tablet_replica.cc:330] T b91a6f3a72fa4f56b62b216afbad693b P f09ff6644f2146e5a4ec517407ff6326: stopping tablet replica
I20260619 18:49:10.138195 15895 raft_consensus.cc:2229] T b91a6f3a72fa4f56b62b216afbad693b P f09ff6644f2146e5a4ec517407ff6326 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:49:10.139411 15895 raft_consensus.cc:2258] T b91a6f3a72fa4f56b62b216afbad693b P f09ff6644f2146e5a4ec517407ff6326 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:49:10.143468 15895 ts_tablet_manager.cc:1905] T b91a6f3a72fa4f56b62b216afbad693b P f09ff6644f2146e5a4ec517407ff6326: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:49:10.150137 15896 tablet_replica.cc:330] T b91a6f3a72fa4f56b62b216afbad693b P 0e48a2270b0f41a3a463e5aa3b3f33cb: stopping tablet replica
I20260619 18:49:10.150767 15896 raft_consensus.cc:2229] T b91a6f3a72fa4f56b62b216afbad693b P 0e48a2270b0f41a3a463e5aa3b3f33cb [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:49:10.151698 15894 tablet_replica.cc:330] T b91a6f3a72fa4f56b62b216afbad693b P 565c231180614ee7986ccd5731aee943: stopping tablet replica
I20260619 18:49:10.152084 15896 raft_consensus.cc:2258] T b91a6f3a72fa4f56b62b216afbad693b P 0e48a2270b0f41a3a463e5aa3b3f33cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:49:10.152482 15894 raft_consensus.cc:2229] T b91a6f3a72fa4f56b62b216afbad693b P 565c231180614ee7986ccd5731aee943 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:49:10.153205 15894 raft_consensus.cc:2258] T b91a6f3a72fa4f56b62b216afbad693b P 565c231180614ee7986ccd5731aee943 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:49:10.155431 15895 ts_tablet_manager.cc:1918] T b91a6f3a72fa4f56b62b216afbad693b P f09ff6644f2146e5a4ec517407ff6326: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:49:10.155844 15895 log.cc:1196] T b91a6f3a72fa4f56b62b216afbad693b P f09ff6644f2146e5a4ec517407ff6326: Deleting WAL directory at /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/minicluster-data/ts-2/wal/wals/b91a6f3a72fa4f56b62b216afbad693b
I20260619 18:49:10.156800 15894 ts_tablet_manager.cc:1905] T b91a6f3a72fa4f56b62b216afbad693b P 565c231180614ee7986ccd5731aee943: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:49:10.156980 15895 ts_tablet_manager.cc:1939] T b91a6f3a72fa4f56b62b216afbad693b P f09ff6644f2146e5a4ec517407ff6326: Deleting consensus metadata
Jun 19 18:49:10 dist-test-slave-m9wg krb5kdc[14885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894945, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.65@KRBTEST.COM
I20260619 18:49:10.159907 15345 catalog_manager.cc:4841] TS f09ff6644f2146e5a4ec517407ff6326 (127.3.243.67:33355): tablet b91a6f3a72fa4f56b62b216afbad693b (table test-table [id=e8cef817dee141f69de9f393587dfd75]) successfully deleted
I20260619 18:49:10.161864 15896 ts_tablet_manager.cc:1905] T b91a6f3a72fa4f56b62b216afbad693b P 0e48a2270b0f41a3a463e5aa3b3f33cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:49:10.171999 15894 ts_tablet_manager.cc:1918] T b91a6f3a72fa4f56b62b216afbad693b P 565c231180614ee7986ccd5731aee943: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:49:10.172394 15894 log.cc:1196] T b91a6f3a72fa4f56b62b216afbad693b P 565c231180614ee7986ccd5731aee943: Deleting WAL directory at /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/minicluster-data/ts-1/wal/wals/b91a6f3a72fa4f56b62b216afbad693b
I20260619 18:49:10.173476 15894 ts_tablet_manager.cc:1939] T b91a6f3a72fa4f56b62b216afbad693b P 565c231180614ee7986ccd5731aee943: Deleting consensus metadata
I20260619 18:49:10.176755 15348 catalog_manager.cc:4841] TS 565c231180614ee7986ccd5731aee943 (127.3.243.66:41791): tablet b91a6f3a72fa4f56b62b216afbad693b (table test-table [id=e8cef817dee141f69de9f393587dfd75]) successfully deleted
I20260619 18:49:10.178260 15867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:49:10.136711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37426 (local address 127.3.243.65:45305)
0619 18:49:10.137176 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0619 18:49:10.137183 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:49:10.140512 (+ 3329us) server_negotiation.cc:407] Connection header received
0619 18:49:10.141012 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:49:10.141017 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:49:10.141249 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:49:10.143964 (+ 2715us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:49:10.144177 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:49:10.145839 (+ 1662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:49:10.151592 (+ 5753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:49:10.152785 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:49:10.152915 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:49:10.161926 (+ 9011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:49:10.162017 (+ 91us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:49:10.162085 (+ 68us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:49:10.162207 (+ 122us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:49:10.171288 (+ 9081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:49:10.173355 (+ 2067us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:49:10.173370 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:49:10.173384 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:49:10.173724 (+ 340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:49:10.174520 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:49:10.174524 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:49:10.174527 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:49:10.175255 (+ 728us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:49:10.175871 (+ 616us) server_negotiation.cc:1035] Waiting for connection context
0619 18:49:10.177430 (+ 1559us) server_negotiation.cc:299] Negotiation successful
0619 18:49:10.177735 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260619 18:49:10.179016 15896 ts_tablet_manager.cc:1918] T b91a6f3a72fa4f56b62b216afbad693b P 0e48a2270b0f41a3a463e5aa3b3f33cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:49:10.179448 15896 log.cc:1196] T b91a6f3a72fa4f56b62b216afbad693b P 0e48a2270b0f41a3a463e5aa3b3f33cb: Deleting WAL directory at /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894810715282-4045-0/minicluster-data/ts-0/wal/wals/b91a6f3a72fa4f56b62b216afbad693b
I20260619 18:49:10.179500 15569 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:49:10.330119 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 15441
I20260619 18:49:10.352639 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 15580
W20260619 18:49:10.352643 15347 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.243.65:45305 (error 108)
I20260619 18:49:10.356499 15900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:49:10.354827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.65:45305 (local address 127.0.0.1:37432)
0619 18:49:10.355553 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0619 18:49:10.355716 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.65:45305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":407,"threads_started":1}
W20260619 18:49:10.357062 15347 catalog_manager.cc:4584] TS 0e48a2270b0f41a3a463e5aa3b3f33cb (127.3.243.65:45305): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b91a6f3a72fa4f56b62b216afbad693b: Network error: Client connection negotiation failed: client connection to 127.3.243.65:45305: connect: Connection refused (error 111)
Jun 19 18:49:10 dist-test-slave-m9wg krb5kdc[14885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894945, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.66@KRBTEST.COM
I20260619 18:49:10.379206 15866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:49:10.353939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51848 (local address 127.3.243.66:41791)
0619 18:49:10.354338 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0619 18:49:10.354344 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:49:10.354372 (+ 28us) server_negotiation.cc:407] Connection header received
0619 18:49:10.354461 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:49:10.354467 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:49:10.354623 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:49:10.354777 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:49:10.357691 (+ 2914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:49:10.358977 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:49:10.362035 (+ 3058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:49:10.363031 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:49:10.363141 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:49:10.369437 (+ 6296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:49:10.369453 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:49:10.369469 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:49:10.369585 (+ 116us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:49:10.374443 (+ 4858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:49:10.375253 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:49:10.375261 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:49:10.375264 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:49:10.375416 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:49:10.376032 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:49:10.376046 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:49:10.376055 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:49:10.376721 (+ 666us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:49:10.377452 (+ 731us) server_negotiation.cc:1035] Waiting for connection context
0619 18:49:10.377742 (+ 290us) server_negotiation.cc:299] Negotiation successful
0619 18:49:10.378435 (+ 693us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164}
I20260619 18:49:10.380213 15712 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:49:10.411432 15900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:49:10.410822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.65:45305 (local address 127.0.0.1:37436)
0619 18:49:10.411117 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0619 18:49:10.411213 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.65:45305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260619 18:49:10.413069 15900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:49:10.412545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.65:45305 (local address 127.0.0.1:37448)
0619 18:49:10.412743 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0619 18:49:10.412815 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.65:45305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260619 18:49:10.474254 15900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:49:10.473586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.65:45305 (local address 127.0.0.1:37452)
0619 18:49:10.473941 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0619 18:49:10.474036 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.65:45305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260619 18:49:10.475668 15900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:49:10.475210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.65:45305 (local address 127.0.0.1:37454)
0619 18:49:10.475364 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0619 18:49:10.475480 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.65:45305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260619 18:49:10.545112 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 15580
I20260619 18:49:10.557062 15900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:49:10.556143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.65:45305 (local address 127.0.0.1:37466)
0619 18:49:10.556503 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0619 18:49:10.556762 (+ 259us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.65:45305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260619 18:49:10.558806 15900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:49:10.558138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.65:45305 (local address 127.0.0.1:37476)
0619 18:49:10.558449 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0619 18:49:10.558653 (+ 204us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.65:45305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260619 18:49:10.564779 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 15723
Jun 19 18:49:10 dist-test-slave-m9wg krb5kdc[14885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894945, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.67@KRBTEST.COM
I20260619 18:49:10.591797 15902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:49:10.566116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37120 (local address 127.3.243.67:33355)
0619 18:49:10.566954 (+ 838us) server_negotiation.cc:206] Beginning negotiation
0619 18:49:10.566958 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:49:10.567029 (+ 71us) server_negotiation.cc:407] Connection header received
0619 18:49:10.567269 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:49:10.567273 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:49:10.567437 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:49:10.567578 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:49:10.569041 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:49:10.570352 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:49:10.572378 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:49:10.573312 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:49:10.573511 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:49:10.582070 (+ 8559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:49:10.582107 (+ 37us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:49:10.582123 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:49:10.582170 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:49:10.587560 (+ 5390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:49:10.588812 (+ 1252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:49:10.588822 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:49:10.588824 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:49:10.589102 (+ 278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:49:10.589646 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:49:10.589649 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:49:10.589651 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:49:10.590464 (+ 813us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:49:10.590824 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0619 18:49:10.591222 (+ 398us) server_negotiation.cc:299] Negotiation successful
0619 18:49:10.591459 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":706,"thread_start_us":401,"threads_started":1}
I20260619 18:49:10.593019 15846 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:49:10.722764 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 15723
I20260619 18:49:10.731861 15900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:49:10.731243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.65:45305 (local address 127.0.0.1:37480)
0619 18:49:10.731491 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0619 18:49:10.731654 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.65:45305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260619 18:49:10.732990 15900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:49:10.732599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.65:45305 (local address 127.0.0.1:37494)
0619 18:49:10.732735 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0619 18:49:10.732818 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.65:45305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260619 18:49:10.736066 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 15329
I20260619 18:49:10.737319 15391 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:49:10.884112 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 15329
I20260619 18:49:10.900976 4045 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 18:49:11.235291 4045 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 18:49:11.235515 4045 mini_ranger.cc:66] Stopping Ranger...
I20260619 18:49:11.586449 4045 mini_ranger.cc:68] Stopped Ranger
2026-06-19 18:49:11.587 UTC [14915] LOG: received smart shutdown request
2026-06-19 18:49:11.591 UTC [14915] LOG: background worker "logical replication launcher" (PID 14924) exited with exit code 1
2026-06-19 18:49:11.592 UTC [14919] LOG: shutting down
2026-06-19 18:49:11.772 UTC [14915] LOG: database system is shut down
2026-06-19T18:49:11Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (47037 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:49:12 dist-test-slave-m9wg krb5kdc[15913](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:49:12 dist-test-slave-m9wg krb5kdc[15913](info): set up 2 sockets
Jun 19 18:49:12 dist-test-slave-m9wg krb5kdc[15913](info): commencing operation
krb5kdc: starting...
W20260619 18:49:13.096436 4045 mini_kdc.cc:121] Time spent starting KDC: real 1.089s user 0.006s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/krb5kdc/test-user.keytab.
Jun 19 18:49:13 dist-test-slave-m9wg krb5kdc[15913](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894953, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:49:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:49:13Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.3.243.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.3.243.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.3.243.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/krb5kdc/rangeradmin_127.3.243.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.3.243.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/krb5kdc/rangeradmin_127.3.243.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.3.243.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.3.243.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.3.243.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/krb5kdc/rangerlookup_127.3.243.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.3.243.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/krb5kdc/rangerlookup_127.3.243.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.243.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.243.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.243.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/krb5kdc/HTTP_127.3.243.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.243.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/krb5kdc/HTTP_127.3.243.84@KRBTEST.COM.keytab.
I20260619 18:49:13.374445 4045 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskRgbpDZ/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/postgres -l logfile start
2026-06-19 18:49:15.493 UTC [15943] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 18:49:15.493 UTC [15943] LOG: listening on IPv4 address "127.3.243.84", port 52775
2026-06-19 18:49:15.496 UTC [15943] LOG: listening on Unix socket "/tmp/.s.PGSQL.52775"
2026-06-19 18:49:15.517 UTC [15946] LOG: database system was shut down at 2026-06-19 18:49:14 UTC
2026-06-19 18:49:15.520 UTC [15943] LOG: database system is ready to accept connections
I20260619 18:49:16.411909 4045 mini_postgres.cc:96] Postgres bound to 52775
2026-06-19 18:49:16.427 UTC [15954] FATAL: database "slave" does not exist
127.3.243.84:52775 - accepting connections
I20260619 18:49:16.427802 4045 mini_ranger.cc:161] Starting Ranger...
I20260619 18:49:16.450230 4045 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 18:49:16.840587 4045 mini_ranger.cc:87] Created ranger Postgres database
I20260619 18:49:16.840698 4045 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ranger-admin
2026-06-19 18:49:17,122 [I] DB FLAVOR :POSTGRES
2026-06-19 18:49:17,123 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 18:49:17,123 [I] Checking connection..
2026-06-19 18:49:17,123 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRgbpDZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRgbpDZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.243.84:52775/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 18:49:17,370 [I] Checking connection passed.
2026-06-19 18:49:17,371 [I] --------- Verifying version history table ---------
2026-06-19 18:49:17,371 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRgbpDZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRgbpDZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.243.84:52775/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 18:49:17,618 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 18:49:17,618 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRgbpDZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRgbpDZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.243.84:52775/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 18:49:17,857 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 18:49:17,857 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 18:49:17,858 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRgbpDZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRgbpDZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.243.84:52775/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskRgbpDZ/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 18:49:18.105 UTC [16018] WARNING: there is no transaction in progress
2026-06-19 18:49:18,110 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 18:49:18,110 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRgbpDZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRgbpDZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.243.84:52775/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 18:49:18,358 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 18:49:18,358 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 18:49:18,358 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRgbpDZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRgbpDZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.243.84:52775/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 18:49:18,606 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRgbpDZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRgbpDZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.243.84:52775/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 18:49:18,839 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRgbpDZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRgbpDZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.243.84:52775/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-m9wg.c.gcp-upstream.internal','N') ;"
2026-06-19 18:49:19,059 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRgbpDZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRgbpDZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.243.84:52775/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 18:49:19,301 [I] Table x_portal_user does not exist in database ranger
2026-06-19 18:49:19,302 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRgbpDZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRgbpDZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.243.84:52775/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 18:49:19,526 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 18:49:19,526 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRgbpDZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRgbpDZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.243.84:52775/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 18:49:19,746 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRgbpDZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRgbpDZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.243.84:52775/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 18:49:19,989 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 18:49:19,989 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRgbpDZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRgbpDZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.243.84:52775/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskRgbpDZ/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 18:49:20.855 UTC [16126] WARNING: there is no transaction in progress
2026-06-19 18:49:20.871 UTC [16126] WARNING: there is no transaction in progress
2026-06-19 18:49:20.886 UTC [16126] WARNING: there is no transaction in progress
2026-06-19 18:49:20.899 UTC [16126] WARNING: there is no transaction in progress
2026-06-19 18:49:20.913 UTC [16126] WARNING: there is no transaction in progress
2026-06-19 18:49:20.928 UTC [16126] WARNING: there is no transaction in progress
2026-06-19 18:49:20.983 UTC [16126] WARNING: there is no transaction in progress
2026-06-19 18:49:20.989 UTC [16126] WARNING: there is no transaction in progress
2026-06-19 18:49:21.002 UTC [16126] WARNING: there is no transaction in progress
2026-06-19 18:49:21.011 UTC [16126] WARNING: there is no transaction in progress
2026-06-19 18:49:21.020 UTC [16126] WARNING: there is no transaction in progress
2026-06-19 18:49:21.033 UTC [16126] WARNING: there is no transaction in progress
2026-06-19 18:49:21.041 UTC [16126] WARNING: there is no transaction in progress
2026-06-19 18:49:21.046 UTC [16126] WARNING: there is no transaction in progress
2026-06-19 18:49:21.052 UTC [16126] WARNING: there is no transaction in progress
2026-06-19 18:49:21.057 UTC [16126] WARNING: there is no transaction in progress
2026-06-19 18:49:21,590 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 18:49:21,591 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRgbpDZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRgbpDZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.243.84:52775/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 18:49:21,822 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 18:49:21,822 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRgbpDZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRgbpDZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.243.84:52775/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 18:49:22,043 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 18:49:22,043 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRgbpDZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRgbpDZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.243.84:52775/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 18:49:22,294 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 18:49:22,294 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRgbpDZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRgbpDZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.243.84:52775/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 18:49:22,533 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRgbpDZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRgbpDZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.243.84:52775/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 18:49:22,780 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRgbpDZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRgbpDZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.243.84:52775/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-m9wg.c.gcp-upstream.internal';"
2026-06-19 18:49:23,005 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 18:49:23,005 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRgbpDZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRgbpDZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.243.84:52775/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 18:49:23,237 [I] DB_PATCHES have already been applied
I20260619 18:49:23.242450 4045 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ranger-admin:/tmp/dist-test-taskRgbpDZ/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskRgbpDZ/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskRgbpDZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRgbpDZ/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 18:49:23.242532 4045 mini_ranger.cc:193] Using host: 127.3.243.84
I20260619 18:49:23.255502 4045 mini_ranger.cc:227] Ranger admin URL: http://127.3.243.84:39909
Jun 19, 2026 6:49:23 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 6:49:23 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 6:49:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 6:49:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 6:49:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 6:49:24 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 6:49:24 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 18:49:24.249467 4045 mini_ranger.cc:160] Time spent starting Ranger: real 7.822s user 0.000s sys 0.035s
Jun 19, 2026 6:49:43 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 6:49:43 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Jun 19, 2026 6:49:43 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 6:49:43 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 6:49:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 6:49:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 6:49:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 6:49:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 6:49:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 6:49:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 6:49:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 6:49:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 6:49:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 6:49:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 6:49:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 6:49:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 6:49:43 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 6:49:44 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 18:49:45.242880 4045 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.3.243.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.3.243.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.3.243.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/krb5kdc/rangerkms_127.3.243.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.3.243.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/krb5kdc/rangerkms_127.3.243.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.243.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.3.243.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.243.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/krb5kdc/HTTP_127.3.243.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.243.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/krb5kdc/HTTP_127.3.243.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 18:49:45 dist-test-slave-m9wg krb5kdc[15913](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894985, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 18:49:45.385216 4045 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 18:49:45.402958 4045 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 18:49:46.238488 4045 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 18:49:46.238610 4045 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ranger-kms
I20260619 18:49:46.238653 4045 mini_ranger_kms.cc:227] Using postgres at 127.3.243.84:52775
2026-06-19 18:49:46,577 [I] DB FLAVOR :POSTGRES
2026-06-19 18:49:46,577 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 18:49:46,577 [I] Checking connection
2026-06-19 18:49:46,577 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRgbpDZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.243.84:52775/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 18:49:46,805 [I] connection success
2026-06-19 18:49:46,805 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 18:49:46,805 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 18:49:46,805 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRgbpDZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.243.84:52775/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 18:49:47,040 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 18:49:47,041 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 18:49:47,041 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 18:49:47,041 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRgbpDZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.243.84:52775/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 18:49:47,283 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 18:49:47.749208 4045 mini_ranger_kms.cc:310] Created kms service
I20260619 18:49:47.998031 4045 mini_ranger_kms.cc:326] Created kudu user
I20260619 18:49:48.043573 4045 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 18:49:48 dist-test-slave-m9wg krb5kdc[15913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894985, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.3.243.84@KRBTEST.COM
I20260619 18:49:48.410887 4045 mini_ranger_kms.cc:384] Added ranger policy
I20260619 18:49:48.411185 4045 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ranger-kms:/tmp/dist-test-taskRgbpDZ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRgbpDZ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskRgbpDZ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 18:49:48.411247 4045 mini_ranger_kms.cc:240] Using host: 127.3.243.84
I20260619 18:49:48.428921 4045 mini_ranger_kms.cc:276] Ranger KMS PID: 16306
I20260619 18:49:48.429028 4045 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.3.243.84:36633
Jun 19, 2026 6:49:49 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 6:49:49 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 6:49:49 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 6:49:49 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 6:49:49 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 6:49:49 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260619 18:49:50.474120 4045 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.089s user 0.057s sys 0.295s
I20260619 18:49:50.474365 4045 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 18:49:50.474537 4045 mini_ranger_kms.cc:401] 127.3.243.84:36633/kms/v1/keys
Jun 19 18:49:50 dist-test-slave-m9wg krb5kdc[15913](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781894990, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.243.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 19 18:49:51 dist-test-slave-m9wg krb5kdc[15913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894990, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.243.84@KRBTEST.COM for HTTP/127.3.243.84@KRBTEST.COM
Jun 19 18:49:51 dist-test-slave-m9wg krb5kdc[15913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894990, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.243.84@KRBTEST.COM for HTTP/127.3.243.84@KRBTEST.COM
Jun 19 18:49:53 dist-test-slave-m9wg krb5kdc[15913](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894993, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.3.243.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab.
Entry for principal kudu/127.3.243.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab.
I20260619 18:49:53.671710 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.126:44203
--webserver_interface=127.3.243.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab
--principal=kudu/127.3.243.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:42157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.126:44203
--ranger_config_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.243.84:36633/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:49:54.022785 16357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:49:54.023061 16357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:49:54.023161 16357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:49:54.030714 16357 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:49:54.030839 16357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:49:54.030908 16357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:49:54.031004 16357 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:49:54.031076 16357 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:49:54.041352 16357 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:42157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/master-0/wal
--ranger_kms_url=127.3.243.84:36633/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.3.243.126:44203
--ranger_config_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab
--principal=kudu/127.3.243.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.126:44203
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.243.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:49:54.042577 16357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:49:54.043869 16357 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:49:54.056628 16362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:49:54.056653 16363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:49:54.057232 16365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:49:54.058255 16357 server_base.cc:1034] running on GCE node
I20260619 18:49:54.059018 16357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:49:54.060475 16357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:49:54.061686 16357 hybrid_clock.cc:648] HybridClock initialized: now 1781894994061643 us; error 37 us; skew 500 ppm
I20260619 18:49:54.062024 16357 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:49:54 dist-test-slave-m9wg krb5kdc[15913](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894994, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:49:54.065333 16357 init.cc:377] Logged in from keytab as kudu/127.3.243.126@KRBTEST.COM (short username kudu)
I20260619 18:49:54.066922 16357 webserver.cc:460] Webserver started at http://127.3.243.126:45275/ using document root <none> and password file <none>
I20260619 18:49:54.067585 16357 fs_manager.cc:359] Metadata directory not provided
I20260619 18:49:54.067709 16357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:49:54.068017 16357 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 18:49:54 dist-test-slave-m9wg krb5kdc[15913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894994, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.126@KRBTEST.COM for HTTP/127.3.243.84@KRBTEST.COM
I20260619 18:49:54.117168 16357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/master-0/data/instance:
uuid: "2bf14fb7b3b3456d910c3d46f4604b37"
format_stamp: "Formatted at 2026-06-19 18:49:54 on dist-test-slave-m9wg"
server_key: "348d140d168b76cf13ed2fda97c50534"
server_key_iv: "ece870cfe1db42eed60288c522fb1383"
server_key_version: "kuduclusterkey@0"
I20260619 18:49:54.118109 16357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/master-0/wal/instance:
uuid: "2bf14fb7b3b3456d910c3d46f4604b37"
format_stamp: "Formatted at 2026-06-19 18:49:54 on dist-test-slave-m9wg"
server_key: "348d140d168b76cf13ed2fda97c50534"
server_key_iv: "ece870cfe1db42eed60288c522fb1383"
server_key_version: "kuduclusterkey@0"
I20260619 18:49:54.123728 16357 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 18:49:54.140504 16372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:49:54.141987 16357 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 18:49:54.142177 16357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/master-0/wal
uuid: "2bf14fb7b3b3456d910c3d46f4604b37"
format_stamp: "Formatted at 2026-06-19 18:49:54 on dist-test-slave-m9wg"
server_key: "348d140d168b76cf13ed2fda97c50534"
server_key_iv: "ece870cfe1db42eed60288c522fb1383"
server_key_version: "kuduclusterkey@0"
I20260619 18:49:54.142396 16357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:49:54.177807 16357 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:49:54.182283 16357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:49:54.182623 16357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:49:54.203037 16357 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.126:44203
I20260619 18:49:54.203126 16423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.126:44203 every 8 connection(s)
I20260619 18:49:54.204468 16357 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
I20260619 18:49:54.208989 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 16357
I20260619 18:49:54.209170 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/master-0/wal/instance
I20260619 18:49:54.209260 16424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:49:54.209859 4045 external_mini_cluster.cc:1428] Setting key 1ea73e273ca15ce539c705f0bdef2f1e
I20260619 18:49:54.224674 16424 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bf14fb7b3b3456d910c3d46f4604b37: Bootstrap starting.
I20260619 18:49:54.229410 16424 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2bf14fb7b3b3456d910c3d46f4604b37: Neither blocks nor log segments found. Creating new log.
I20260619 18:49:54.230966 16424 log.cc:824] T 00000000000000000000000000000000 P 2bf14fb7b3b3456d910c3d46f4604b37: Log is configured to *not* fsync() on all Append() calls
I20260619 18:49:54.234549 16424 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bf14fb7b3b3456d910c3d46f4604b37: No bootstrap required, opened a new log
Jun 19 18:49:54 dist-test-slave-m9wg krb5kdc[15913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894993, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.126@KRBTEST.COM
I20260619 18:49:54.246013 16427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:49:54.211911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53388 (local address 127.3.243.126:44203)
0619 18:49:54.226692 (+ 14781us) server_negotiation.cc:206] Beginning negotiation
0619 18:49:54.226705 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 18:49:54.226753 (+ 48us) server_negotiation.cc:407] Connection header received
0619 18:49:54.228827 (+ 2074us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:49:54.228849 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:49:54.229242 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:49:54.229605 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:49:54.230995 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:49:54.232838 (+ 1843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:49:54.234228 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:49:54.234998 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:49:54.235170 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:49:54.238805 (+ 3635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:49:54.238817 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:49:54.238830 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:49:54.238877 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:49:54.242049 (+ 3172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:49:54.242652 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:49:54.242657 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:49:54.242663 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:49:54.242833 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:49:54.243194 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:49:54.243198 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:49:54.243200 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:49:54.243744 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:49:54.244032 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0619 18:49:54.244492 (+ 460us) server_negotiation.cc:299] Negotiation successful
0619 18:49:54.244904 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14537,"thread_start_us":15370,"threads_started":1}
I20260619 18:49:54.246321 16424 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2bf14fb7b3b3456d910c3d46f4604b37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bf14fb7b3b3456d910c3d46f4604b37" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 44203 } }
I20260619 18:49:54.246784 16424 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2bf14fb7b3b3456d910c3d46f4604b37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:49:54.246917 16424 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2bf14fb7b3b3456d910c3d46f4604b37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bf14fb7b3b3456d910c3d46f4604b37, State: Initialized, Role: FOLLOWER
I20260619 18:49:54.247690 16424 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2bf14fb7b3b3456d910c3d46f4604b37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bf14fb7b3b3456d910c3d46f4604b37" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 44203 } }
I20260619 18:49:54.247994 16424 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2bf14fb7b3b3456d910c3d46f4604b37 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:49:54.248186 16424 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2bf14fb7b3b3456d910c3d46f4604b37 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:49:54.248464 16424 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2bf14fb7b3b3456d910c3d46f4604b37 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:49:54.251948 16424 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2bf14fb7b3b3456d910c3d46f4604b37 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bf14fb7b3b3456d910c3d46f4604b37" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 44203 } }
I20260619 18:49:54.252565 16424 leader_election.cc:302] T 00000000000000000000000000000000 P 2bf14fb7b3b3456d910c3d46f4604b37 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2bf14fb7b3b3456d910c3d46f4604b37; no voters:
I20260619 18:49:54.253234 16424 leader_election.cc:288] T 00000000000000000000000000000000 P 2bf14fb7b3b3456d910c3d46f4604b37 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:49:54.253324 16429 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2bf14fb7b3b3456d910c3d46f4604b37 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:49:54.255494 16424 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2bf14fb7b3b3456d910c3d46f4604b37 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:49:54.260421 16429 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2bf14fb7b3b3456d910c3d46f4604b37 [term 1 LEADER]: Becoming Leader. State: Replica: 2bf14fb7b3b3456d910c3d46f4604b37, State: Running, Role: LEADER
I20260619 18:49:54.261010 16429 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2bf14fb7b3b3456d910c3d46f4604b37 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bf14fb7b3b3456d910c3d46f4604b37" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 44203 } }
I20260619 18:49:54.263821 16430 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bf14fb7b3b3456d910c3d46f4604b37 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2bf14fb7b3b3456d910c3d46f4604b37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bf14fb7b3b3456d910c3d46f4604b37" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 44203 } } }
I20260619 18:49:54.264262 16430 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bf14fb7b3b3456d910c3d46f4604b37 [sys.catalog]: This master's current role is: LEADER
I20260619 18:49:54.264840 16431 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bf14fb7b3b3456d910c3d46f4604b37 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2bf14fb7b3b3456d910c3d46f4604b37. Latest consensus state: current_term: 1 leader_uuid: "2bf14fb7b3b3456d910c3d46f4604b37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bf14fb7b3b3456d910c3d46f4604b37" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 44203 } } }
I20260619 18:49:54.265159 16431 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bf14fb7b3b3456d910c3d46f4604b37 [sys.catalog]: This master's current role is: LEADER
I20260619 18:49:54.268108 16436 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:49:54.273695 16436 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:49:54.277693 16424 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 18:49:54.291775 16436 catalog_manager.cc:1347] Generated new cluster ID: d83e853089274f1282da7102ac311463
I20260619 18:49:54.291926 16436 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:49:54.320875 16436 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:49:54.322860 16436 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:49:54.332443 16436 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2bf14fb7b3b3456d910c3d46f4604b37: Generated new TSK 0
I20260619 18:49:54.333868 16436 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.243.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.65.keytab.
Entry for principal kudu/127.3.243.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.65.keytab.
I20260619 18:49:55.075196 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.243.84:36633/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.65:0
--local_ip_for_outbound_sockets=127.3.243.65
--webserver_interface=127.3.243.65
--webserver_port=0
--tserver_master_addrs=127.3.243.126:44203
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.65.keytab
--principal=kudu/127.3.243.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:42157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 18:49:55 dist-test-slave-m9wg krb5kdc[15913](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781894995, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260619 18:49:55.480444 16469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:49:55.480722 16469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:49:55.480857 16469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:49:55.489357 16469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:49:55.489569 16469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:49:55.489813 16469 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.65
I20260619 18:49:55.501554 16469 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:42157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.3.243.84:36633/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.65.keytab
--principal=kudu/127.3.243.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.243.65
--webserver_port=0
--tserver_master_addrs=127.3.243.126:44203
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.65
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:49:55.503423 16469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:49:55.504819 16469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:49:55.525115 16479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:49:55.525928 16480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:49:55.529719 16482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:49:55.530937 16469 server_base.cc:1034] running on GCE node
I20260619 18:49:55.531622 16469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:49:55.532742 16469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:49:55.534001 16469 hybrid_clock.cc:648] HybridClock initialized: now 1781894995533931 us; error 51 us; skew 500 ppm
I20260619 18:49:55.534348 16469 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:49:55 dist-test-slave-m9wg krb5kdc[15913](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894995, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:49:55.537740 16469 init.cc:377] Logged in from keytab as kudu/127.3.243.65@KRBTEST.COM (short username kudu)
I20260619 18:49:55.540525 16469 webserver.cc:460] Webserver started at http://127.3.243.65:45311/ using document root <none> and password file <none>
I20260619 18:49:55.541287 16469 fs_manager.cc:359] Metadata directory not provided
I20260619 18:49:55.541420 16469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:49:55.541749 16469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 18:49:55 dist-test-slave-m9wg krb5kdc[15913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894995, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.65@KRBTEST.COM for HTTP/127.3.243.84@KRBTEST.COM
I20260619 18:49:55.567333 16469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ts-0/data/instance:
uuid: "903297c2967c48f0a501243b94829f0d"
format_stamp: "Formatted at 2026-06-19 18:49:55 on dist-test-slave-m9wg"
server_key: "110b065ea475454f77babd6b10a2c627"
server_key_iv: "067e8ae792b0c8a7f35d0775aefdc17a"
server_key_version: "kuduclusterkey@0"
I20260619 18:49:55.570604 16469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ts-0/wal/instance:
uuid: "903297c2967c48f0a501243b94829f0d"
format_stamp: "Formatted at 2026-06-19 18:49:55 on dist-test-slave-m9wg"
server_key: "110b065ea475454f77babd6b10a2c627"
server_key_iv: "067e8ae792b0c8a7f35d0775aefdc17a"
server_key_version: "kuduclusterkey@0"
I20260619 18:49:55.577375 16469 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260619 18:49:55.616999 16489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:49:55.619107 16469 fs_manager.cc:725] Time spent opening block manager: real 0.021s user 0.006s sys 0.004s
I20260619 18:49:55.619427 16469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ts-0/wal
uuid: "903297c2967c48f0a501243b94829f0d"
format_stamp: "Formatted at 2026-06-19 18:49:55 on dist-test-slave-m9wg"
server_key: "110b065ea475454f77babd6b10a2c627"
server_key_iv: "067e8ae792b0c8a7f35d0775aefdc17a"
server_key_version: "kuduclusterkey@0"
I20260619 18:49:55.619828 16469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:49:55.643512 16469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:49:55.648764 16469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:49:55.649155 16469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:49:55.650547 16469 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:49:55.652498 16469 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:49:55.652578 16469 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:49:55.652720 16469 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:49:55.652791 16469 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:49:55.750052 16469 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.65:39543
I20260619 18:49:55.750065 16601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.65:39543 every 8 connection(s)
I20260619 18:49:55.751361 16469 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ts-0/data/info.pb
I20260619 18:49:55.757315 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 16469
I20260619 18:49:55.757546 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ts-0/wal/instance
I20260619 18:49:55.758386 4045 external_mini_cluster.cc:1428] Setting key 3b212c748e5f6f655d9097413a88ec0d
Jun 19 18:49:55 dist-test-slave-m9wg krb5kdc[15913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894995, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.65@KRBTEST.COM for kudu/127.3.243.126@KRBTEST.COM
I20260619 18:49:55.796322 16605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:49:55.756159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.65:52275 (local address 127.3.243.126:44203)
0619 18:49:55.763882 (+ 7723us) server_negotiation.cc:206] Beginning negotiation
0619 18:49:55.763888 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:49:55.763906 (+ 18us) server_negotiation.cc:407] Connection header received
0619 18:49:55.763961 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:49:55.763966 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:49:55.764045 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:49:55.764185 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:49:55.770863 (+ 6678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:49:55.772073 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:49:55.774615 (+ 2542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:49:55.775351 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:49:55.775448 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:49:55.787633 (+ 12185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:49:55.787641 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:49:55.787646 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:49:55.787706 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:49:55.791526 (+ 3820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:49:55.793141 (+ 1615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:49:55.793147 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:49:55.793151 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:49:55.793327 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:49:55.794177 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:49:55.794182 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:49:55.794186 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:49:55.794527 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:49:55.794668 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 18:49:55.795623 (+ 955us) server_negotiation.cc:299] Negotiation successful
0619 18:49:55.795862 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7604,"thread_start_us":7466,"threads_started":1}
I20260619 18:49:55.797616 16604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:49:55.755400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:44203 (local address 127.3.243.65:52275)
0619 18:49:55.760752 (+ 5352us) negotiation.cc:107] Waiting for socket to connect
0619 18:49:55.760805 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 18:49:55.763223 (+ 2418us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:49:55.769045 (+ 5822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:49:55.769059 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:49:55.769745 (+ 686us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:49:55.770614 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:49:55.770639 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:49:55.772301 (+ 1662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:49:55.772307 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:49:55.774438 (+ 2131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:49:55.774455 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:49:55.775843 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:49:55.775850 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:49:55.776140 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:49:55.776939 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:49:55.776970 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:49:55.779873 (+ 2903us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:49:55.792279 (+ 12406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:49:55.792291 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:49:55.792311 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:49:55.792978 (+ 667us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:49:55.793756 (+ 778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:49:55.793762 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:49:55.793766 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:49:55.794055 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:49:55.794681 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:49:55.794689 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:49:55.795180 (+ 491us) client_negotiation.cc:769] Sending connection context
0619 18:49:55.795508 (+ 328us) client_negotiation.cc:240] Negotiation successful
0619 18:49:55.796123 (+ 615us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5010,"thread_start_us":4778,"threads_started":1}
I20260619 18:49:55.799481 16602 heartbeater.cc:347] Connected to a master server at 127.3.243.126:44203
I20260619 18:49:55.799839 16602 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.3.243.66@KRBTEST.COM; defaulting to no policy
I20260619 18:49:55.800776 16602 heartbeater.cc:511] Master 127.3.243.126:44203 requested a full tablet report, sending...
I20260619 18:49:55.803326 16389 ts_manager.cc:194] Registered new tserver with Master: 903297c2967c48f0a501243b94829f0d (127.3.243.65:39543)
I20260619 18:49:55.805442 16389 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.65@KRBTEST.COM'} at 127.3.243.65:52275
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.66@KRBTEST.COM" created.
Jun 19 18:49:55 dist-test-slave-m9wg krb5kdc[15913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894995, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.126@KRBTEST.COM for HTTP/127.3.243.84@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.66.keytab.
Entry for principal kudu/127.3.243.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.66.keytab.
I20260619 18:49:55.858897 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.243.84:36633/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.66:0
--local_ip_for_outbound_sockets=127.3.243.66
--webserver_interface=127.3.243.66
--webserver_port=0
--tserver_master_addrs=127.3.243.126:44203
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.66.keytab
--principal=kudu/127.3.243.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:42157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 18:49:56 dist-test-slave-m9wg krb5kdc[15913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894995, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.126@KRBTEST.COM for HTTP/127.3.243.84@KRBTEST.COM
W20260619 18:49:56.367093 16608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:49:56.367554 16608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:49:56.367762 16608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:49:56.379588 16608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:49:56.379925 16608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:49:56.380198 16608 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.66
I20260619 18:49:56.396378 16608 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:42157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.3.243.84:36633/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.66.keytab
--principal=kudu/127.3.243.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.243.66
--webserver_port=0
--tserver_master_addrs=127.3.243.126:44203
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.66
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:49:56.398412 16608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:49:56.400442 16608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:49:56.444551 16616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:49:56.447187 16619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:49:56.447958 16615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:49:56.448783 16608 server_base.cc:1034] running on GCE node
I20260619 18:49:56.449699 16608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:49:56.462175 16608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:49:56.473258 16608 hybrid_clock.cc:648] HybridClock initialized: now 1781894996473081 us; error 183 us; skew 500 ppm
I20260619 18:49:56.473784 16608 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:49:56 dist-test-slave-m9wg krb5kdc[15913](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894996, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:49:56.478420 16608 init.cc:377] Logged in from keytab as kudu/127.3.243.66@KRBTEST.COM (short username kudu)
I20260619 18:49:56.493045 16608 webserver.cc:460] Webserver started at http://127.3.243.66:41495/ using document root <none> and password file <none>
I20260619 18:49:56.493782 16608 fs_manager.cc:359] Metadata directory not provided
I20260619 18:49:56.493902 16608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:49:56.494220 16608 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 18:49:56 dist-test-slave-m9wg krb5kdc[15913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894996, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.66@KRBTEST.COM for HTTP/127.3.243.84@KRBTEST.COM
I20260619 18:49:56.514569 16608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ts-1/data/instance:
uuid: "fb9191a13d5d4605aa972c67ee2c6f0d"
format_stamp: "Formatted at 2026-06-19 18:49:56 on dist-test-slave-m9wg"
server_key: "20a81610412617f4b5f9530c8d3171fe"
server_key_iv: "f03fff718b484eda37c17d3398308a46"
server_key_version: "kuduclusterkey@0"
I20260619 18:49:56.515470 16608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ts-1/wal/instance:
uuid: "fb9191a13d5d4605aa972c67ee2c6f0d"
format_stamp: "Formatted at 2026-06-19 18:49:56 on dist-test-slave-m9wg"
server_key: "20a81610412617f4b5f9530c8d3171fe"
server_key_iv: "f03fff718b484eda37c17d3398308a46"
server_key_version: "kuduclusterkey@0"
I20260619 18:49:56.522192 16608 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.009s sys 0.000s
I20260619 18:49:56.553275 16632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:49:56.554876 16608 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260619 18:49:56.555074 16608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ts-1/wal
uuid: "fb9191a13d5d4605aa972c67ee2c6f0d"
format_stamp: "Formatted at 2026-06-19 18:49:56 on dist-test-slave-m9wg"
server_key: "20a81610412617f4b5f9530c8d3171fe"
server_key_iv: "f03fff718b484eda37c17d3398308a46"
server_key_version: "kuduclusterkey@0"
I20260619 18:49:56.555308 16608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:49:56.575601 16608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:49:56.579998 16608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:49:56.580351 16608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:49:56.581529 16608 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:49:56.583271 16608 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:49:56.583357 16608 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:49:56.583528 16608 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:49:56.583595 16608 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:49:56.644197 16608 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.66:35753
I20260619 18:49:56.644229 16744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.66:35753 every 8 connection(s)
I20260619 18:49:56.646075 16608 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ts-1/data/info.pb
I20260619 18:49:56.654410 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 16608
I20260619 18:49:56.654616 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ts-1/wal/instance
I20260619 18:49:56.655261 4045 external_mini_cluster.cc:1428] Setting key 0a823c3a6b0c3dde9fd37926a71b5bd4
Jun 19 18:49:56 dist-test-slave-m9wg krb5kdc[15913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894996, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.66@KRBTEST.COM for kudu/127.3.243.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.243.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.67@KRBTEST.COM" created.
I20260619 18:49:56.684862 16748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:49:56.650380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.66:33055 (local address 127.3.243.126:44203)
0619 18:49:56.660477 (+ 10097us) server_negotiation.cc:206] Beginning negotiation
0619 18:49:56.660483 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:49:56.660501 (+ 18us) server_negotiation.cc:407] Connection header received
0619 18:49:56.660574 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:49:56.660579 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:49:56.660645 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:49:56.660738 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:49:56.668649 (+ 7911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:49:56.669771 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:49:56.672247 (+ 2476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:49:56.672885 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:49:56.673005 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:49:56.677431 (+ 4426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:49:56.677439 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:49:56.677444 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:49:56.677487 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:49:56.681153 (+ 3666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:49:56.682050 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:49:56.682056 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:49:56.682060 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:49:56.682210 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:49:56.682753 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:49:56.682757 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:49:56.682759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:49:56.683048 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:49:56.683160 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 18:49:56.684257 (+ 1097us) server_negotiation.cc:299] Negotiation successful
0619 18:49:56.684489 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9983,"thread_start_us":14607,"threads_started":1}
I20260619 18:49:56.686074 16747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:49:56.649890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:44203 (local address 127.3.243.66:33055)
0619 18:49:56.656889 (+ 6999us) negotiation.cc:107] Waiting for socket to connect
0619 18:49:56.656959 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0619 18:49:56.659660 (+ 2701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:49:56.666668 (+ 7008us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:49:56.666681 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:49:56.667411 (+ 730us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:49:56.668422 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:49:56.668445 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:49:56.669936 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:49:56.669941 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:49:56.672068 (+ 2127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:49:56.672094 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:49:56.673145 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:49:56.673151 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:49:56.673418 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:49:56.674304 (+ 886us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:49:56.674333 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:49:56.677155 (+ 2822us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:49:56.681307 (+ 4152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:49:56.681316 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:49:56.681335 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:49:56.681914 (+ 579us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:49:56.682317 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:49:56.682322 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:49:56.682326 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:49:56.682646 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:49:56.683185 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:49:56.683193 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:49:56.683715 (+ 522us) client_negotiation.cc:769] Sending connection context
0619 18:49:56.684063 (+ 348us) client_negotiation.cc:240] Negotiation successful
0619 18:49:56.684654 (+ 591us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6619,"thread_start_us":14156,"threads_started":1}
I20260619 18:49:56.687546 16745 heartbeater.cc:347] Connected to a master server at 127.3.243.126:44203
I20260619 18:49:56.687876 16745 heartbeater.cc:464] Registering TS with master...
I20260619 18:49:56.688740 16745 heartbeater.cc:511] Master 127.3.243.126:44203 requested a full tablet report, sending...
I20260619 18:49:56.690577 16389 ts_manager.cc:194] Registered new tserver with Master: fb9191a13d5d4605aa972c67ee2c6f0d (127.3.243.66:35753)
I20260619 18:49:56.692962 16389 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.66@KRBTEST.COM'} at 127.3.243.66:33055
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.67.keytab.
Entry for principal kudu/127.3.243.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.67.keytab.
I20260619 18:49:56.715194 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.243.84:36633/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.67:0
--local_ip_for_outbound_sockets=127.3.243.67
--webserver_interface=127.3.243.67
--webserver_port=0
--tserver_master_addrs=127.3.243.126:44203
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.67.keytab
--principal=kudu/127.3.243.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:42157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 18:49:56.809504 16602 heartbeater.cc:503] Master 127.3.243.126:44203 was elected leader, sending a full tablet report...
W20260619 18:49:57.070807 16751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:49:57.071085 16751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:49:57.071188 16751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:49:57.078380 16751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:49:57.078485 16751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:49:57.078600 16751 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.67
I20260619 18:49:57.088291 16751 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:42157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.3.243.84:36633/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.67.keytab
--principal=kudu/127.3.243.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.243.67
--webserver_port=0
--tserver_master_addrs=127.3.243.126:44203
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.67
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:49:57.089526 16751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:49:57.090797 16751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:49:57.103775 16757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:49:57.103859 16756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:49:57.103783 16759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:49:57.104683 16751 server_base.cc:1034] running on GCE node
I20260619 18:49:57.105216 16751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:49:57.106122 16751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:49:57.107340 16751 hybrid_clock.cc:648] HybridClock initialized: now 1781894997107280 us; error 43 us; skew 500 ppm
I20260619 18:49:57.107664 16751 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:49:57 dist-test-slave-m9wg krb5kdc[15913](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894997, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:49:57.113984 16751 init.cc:377] Logged in from keytab as kudu/127.3.243.67@KRBTEST.COM (short username kudu)
I20260619 18:49:57.115458 16751 webserver.cc:460] Webserver started at http://127.3.243.67:37917/ using document root <none> and password file <none>
I20260619 18:49:57.116120 16751 fs_manager.cc:359] Metadata directory not provided
I20260619 18:49:57.116291 16751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:49:57.116595 16751 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 18:49:57 dist-test-slave-m9wg krb5kdc[15913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894997, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.67@KRBTEST.COM for HTTP/127.3.243.84@KRBTEST.COM
I20260619 18:49:57.135106 16751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ts-2/data/instance:
uuid: "0f9b804b0e4d4d98978b08e172a9d104"
format_stamp: "Formatted at 2026-06-19 18:49:57 on dist-test-slave-m9wg"
server_key: "050b36ea704407cfae3570c5a851c7c0"
server_key_iv: "40b3f740ac975a0e57699fb3429423a4"
server_key_version: "kuduclusterkey@0"
I20260619 18:49:57.136310 16751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ts-2/wal/instance:
uuid: "0f9b804b0e4d4d98978b08e172a9d104"
format_stamp: "Formatted at 2026-06-19 18:49:57 on dist-test-slave-m9wg"
server_key: "050b36ea704407cfae3570c5a851c7c0"
server_key_iv: "40b3f740ac975a0e57699fb3429423a4"
server_key_version: "kuduclusterkey@0"
I20260619 18:49:57.142313 16751 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260619 18:49:57.160228 16766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:49:57.161998 16751 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260619 18:49:57.162187 16751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ts-2/wal
uuid: "0f9b804b0e4d4d98978b08e172a9d104"
format_stamp: "Formatted at 2026-06-19 18:49:57 on dist-test-slave-m9wg"
server_key: "050b36ea704407cfae3570c5a851c7c0"
server_key_iv: "40b3f740ac975a0e57699fb3429423a4"
server_key_version: "kuduclusterkey@0"
I20260619 18:49:57.162454 16751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:49:57.205675 16751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:49:57.209800 16751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:49:57.210111 16751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:49:57.211134 16751 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:49:57.212633 16751 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:49:57.212713 16751 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:49:57.212842 16751 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:49:57.212908 16751 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:49:57.248252 16751 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.67:33993
I20260619 18:49:57.248291 16878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.67:33993 every 8 connection(s)
I20260619 18:49:57.249480 16751 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ts-2/data/info.pb
I20260619 18:49:57.257076 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 16751
I20260619 18:49:57.257223 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ts-2/wal/instance
I20260619 18:49:57.257637 4045 external_mini_cluster.cc:1428] Setting key 2f211cc05a6e2de5841f5aef827bedea
Jun 19 18:49:57 dist-test-slave-m9wg krb5kdc[15913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894997, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.67@KRBTEST.COM for kudu/127.3.243.126@KRBTEST.COM
I20260619 18:49:57.268910 16881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:49:57.251811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.67:42379 (local address 127.3.243.126:44203)
0619 18:49:57.252278 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0619 18:49:57.252282 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:49:57.253259 (+ 977us) server_negotiation.cc:407] Connection header received
0619 18:49:57.254974 (+ 1715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:49:57.254981 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:49:57.255057 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:49:57.255151 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:49:57.256686 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:49:57.257484 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:49:57.259247 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:49:57.259766 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:49:57.259902 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:49:57.263177 (+ 3275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:49:57.263182 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:49:57.263185 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:49:57.263226 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:49:57.265843 (+ 2617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:49:57.266543 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:49:57.266547 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:49:57.266549 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:49:57.266685 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:49:57.267139 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:49:57.267142 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:49:57.267144 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:49:57.267423 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:49:57.267568 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 18:49:57.268265 (+ 697us) server_negotiation.cc:299] Negotiation successful
0619 18:49:57.268525 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":218,"threads_started":1}
I20260619 18:49:57.269840 16882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:49:57.252456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:44203 (local address 127.3.243.67:42379)
0619 18:49:57.253100 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0619 18:49:57.253143 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 18:49:57.254706 (+ 1563us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:49:57.255408 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:49:57.255416 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:49:57.255867 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:49:57.256517 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:49:57.256535 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:49:57.257620 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:49:57.257623 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:49:57.259114 (+ 1491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:49:57.259126 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:49:57.259956 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:49:57.259961 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:49:57.260205 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:49:57.260895 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:49:57.260917 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:49:57.262963 (+ 2046us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:49:57.265970 (+ 3007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:49:57.265979 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:49:57.265995 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:49:57.266425 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:49:57.266786 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:49:57.266791 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:49:57.266795 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:49:57.267029 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:49:57.267565 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:49:57.267572 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:49:57.267890 (+ 318us) client_negotiation.cc:769] Sending connection context
0619 18:49:57.268116 (+ 226us) client_negotiation.cc:240] Negotiation successful
0619 18:49:57.268661 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":261,"threads_started":1}
I20260619 18:49:57.271246 16879 heartbeater.cc:347] Connected to a master server at 127.3.243.126:44203
I20260619 18:49:57.271567 16879 heartbeater.cc:464] Registering TS with master...
I20260619 18:49:57.272460 16879 heartbeater.cc:511] Master 127.3.243.126:44203 requested a full tablet report, sending...
I20260619 18:49:57.273957 16389 ts_manager.cc:194] Registered new tserver with Master: 0f9b804b0e4d4d98978b08e172a9d104 (127.3.243.67:33993)
I20260619 18:49:57.274767 16389 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.67@KRBTEST.COM'} at 127.3.243.67:42379
I20260619 18:49:57.285419 4045 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 18:49:57.287002 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 16469
Jun 19 18:49:57 dist-test-slave-m9wg krb5kdc[15913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894993, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.65@KRBTEST.COM
I20260619 18:49:57.303905 16884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:49:57.287931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54212 (local address 127.3.243.65:39543)
0619 18:49:57.288577 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0619 18:49:57.288584 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:49:57.288609 (+ 25us) server_negotiation.cc:407] Connection header received
0619 18:49:57.288682 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:49:57.288688 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:49:57.288889 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:49:57.289021 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:49:57.290528 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:49:57.291684 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:49:57.293375 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:49:57.293970 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:49:57.294073 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:49:57.298365 (+ 4292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:49:57.298379 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:49:57.298397 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:49:57.298440 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:49:57.301284 (+ 2844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:49:57.301947 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:49:57.301953 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:49:57.301955 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:49:57.302125 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:49:57.302503 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:49:57.302511 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:49:57.302514 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:49:57.302882 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:49:57.303140 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0619 18:49:57.303303 (+ 163us) server_negotiation.cc:299] Negotiation successful
0619 18:49:57.303511 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":501,"thread_start_us":363,"threads_started":1}
I20260619 18:49:57.305269 16597 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:49:57.387053 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 16469
I20260619 18:49:57.399377 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 16608
Jun 19 18:49:57 dist-test-slave-m9wg krb5kdc[15913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894993, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.66@KRBTEST.COM
I20260619 18:49:57.414930 16886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:49:57.400286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48884 (local address 127.3.243.66:35753)
0619 18:49:57.400837 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0619 18:49:57.400843 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:49:57.400868 (+ 25us) server_negotiation.cc:407] Connection header received
0619 18:49:57.400927 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:49:57.400933 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:49:57.401153 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:49:57.401314 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:49:57.402436 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:49:57.403414 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:49:57.404817 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:49:57.405368 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:49:57.405468 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:49:57.409786 (+ 4318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:49:57.409798 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:49:57.409812 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:49:57.409846 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:49:57.412513 (+ 2667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:49:57.413259 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:49:57.413266 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:49:57.413268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:49:57.413396 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:49:57.413681 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:49:57.413684 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:49:57.413685 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:49:57.413998 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:49:57.414220 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0619 18:49:57.414447 (+ 227us) server_negotiation.cc:299] Negotiation successful
0619 18:49:57.414647 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":267,"threads_started":1}
I20260619 18:49:57.416260 16740 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:49:57.497605 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 16608
I20260619 18:49:57.510003 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 16751
Jun 19 18:49:57 dist-test-slave-m9wg krb5kdc[15913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894993, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.67@KRBTEST.COM
I20260619 18:49:57.525162 16888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:49:57.510866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34692 (local address 127.3.243.67:33993)
0619 18:49:57.511525 (+ 659us) server_negotiation.cc:206] Beginning negotiation
0619 18:49:57.511543 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 18:49:57.511572 (+ 29us) server_negotiation.cc:407] Connection header received
0619 18:49:57.511647 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:49:57.511656 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:49:57.511912 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:49:57.512081 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:49:57.513332 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:49:57.514353 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:49:57.515562 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:49:57.516099 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:49:57.516243 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:49:57.520197 (+ 3954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:49:57.520213 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:49:57.520229 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:49:57.520263 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:49:57.522709 (+ 2446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:49:57.523306 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:49:57.523312 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:49:57.523314 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:49:57.523436 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:49:57.523719 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:49:57.523722 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:49:57.523724 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:49:57.524040 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:49:57.524376 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0619 18:49:57.524694 (+ 318us) server_negotiation.cc:299] Negotiation successful
0619 18:49:57.524888 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":345,"threads_started":1}
I20260619 18:49:57.526401 16874 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:49:57.608247 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 16751
I20260619 18:49:57.620229 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 16357
I20260619 18:49:57.621472 16419 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:49:57.728232 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 16357
I20260619 18:49:57.745821 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.126:44203
--webserver_interface=127.3.243.126
--webserver_port=45275
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab
--principal=kudu/127.3.243.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:42157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.126:44203
--ranger_config_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.243.84:36633/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.243.84:36633/kms with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:49:58.057313 16891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:49:58.057600 16891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:49:58.057724 16891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:49:58.065268 16891 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:49:58.065375 16891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:49:58.065430 16891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:49:58.065507 16891 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:49:58.065567 16891 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:49:58.075722 16891 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:42157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.243.84:36633/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.3.243.126:44203
--ranger_config_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab
--principal=kudu/127.3.243.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.126:44203
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.243.126
--webserver_port=45275
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:49:58.077008 16891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:49:58.078296 16891 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:49:58.089241 16896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:49:58.089214 16897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:49:58.089274 16899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:49:58.090107 16891 server_base.cc:1034] running on GCE node
I20260619 18:49:58.090818 16891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:49:58.092141 16891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:49:58.093376 16891 hybrid_clock.cc:648] HybridClock initialized: now 1781894998093339 us; error 32 us; skew 500 ppm
I20260619 18:49:58.093713 16891 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:49:58 dist-test-slave-m9wg krb5kdc[15913](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894998, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:49:58.096724 16891 init.cc:377] Logged in from keytab as kudu/127.3.243.126@KRBTEST.COM (short username kudu)
I20260619 18:49:58.098107 16891 webserver.cc:460] Webserver started at http://127.3.243.126:45275/ using document root <none> and password file <none>
I20260619 18:49:58.098738 16891 fs_manager.cc:359] Metadata directory not provided
I20260619 18:49:58.098856 16891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 18:49:58 dist-test-slave-m9wg krb5kdc[15913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894998, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.126@KRBTEST.COM for HTTP/127.3.243.84@KRBTEST.COM
I20260619 18:49:58.150905 16891 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 18:49:58.154099 16907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:49:58.155591 16891 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 18:49:58.155779 16891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/master-0/wal
uuid: "2bf14fb7b3b3456d910c3d46f4604b37"
format_stamp: "Formatted at 2026-06-19 18:49:54 on dist-test-slave-m9wg"
server_key: "348d140d168b76cf13ed2fda97c50534"
server_key_iv: "ece870cfe1db42eed60288c522fb1383"
server_key_version: "kuduclusterkey@0"
I20260619 18:49:58.156795 16891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:49:58.172719 16891 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:49:58.176731 16891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:49:58.176997 16891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:49:58.193688 16891 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.126:44203
I20260619 18:49:58.193710 16958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.126:44203 every 8 connection(s)
I20260619 18:49:58.195163 16891 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
I20260619 18:49:58.197278 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 16891
I20260619 18:49:58.198282 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.243.84:36633/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.65:39543
--local_ip_for_outbound_sockets=127.3.243.65
--tserver_master_addrs=127.3.243.126:44203
--webserver_port=45311
--webserver_interface=127.3.243.65
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.65.keytab
--principal=kudu/127.3.243.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:42157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.243.84:36633/kms with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 18:49:58.203866 16959 sys_catalog.cc:263] Verifying existing consensus state
I20260619 18:49:58.207861 16959 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bf14fb7b3b3456d910c3d46f4604b37: Bootstrap starting.
I20260619 18:49:58.225957 16959 log.cc:824] T 00000000000000000000000000000000 P 2bf14fb7b3b3456d910c3d46f4604b37: Log is configured to *not* fsync() on all Append() calls
I20260619 18:49:58.234462 16959 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bf14fb7b3b3456d910c3d46f4604b37: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 18:49:58.234977 16959 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bf14fb7b3b3456d910c3d46f4604b37: Bootstrap complete.
I20260619 18:49:58.243573 16959 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2bf14fb7b3b3456d910c3d46f4604b37 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bf14fb7b3b3456d910c3d46f4604b37" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 44203 } }
I20260619 18:49:58.244424 16959 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2bf14fb7b3b3456d910c3d46f4604b37 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bf14fb7b3b3456d910c3d46f4604b37, State: Initialized, Role: FOLLOWER
I20260619 18:49:58.245046 16959 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2bf14fb7b3b3456d910c3d46f4604b37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bf14fb7b3b3456d910c3d46f4604b37" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 44203 } }
I20260619 18:49:58.245262 16959 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2bf14fb7b3b3456d910c3d46f4604b37 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:49:58.245370 16959 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2bf14fb7b3b3456d910c3d46f4604b37 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:49:58.245532 16959 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2bf14fb7b3b3456d910c3d46f4604b37 [term 1 FOLLOWER]: Advancing to term 2
I20260619 18:49:58.249199 16959 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2bf14fb7b3b3456d910c3d46f4604b37 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bf14fb7b3b3456d910c3d46f4604b37" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 44203 } }
I20260619 18:49:58.249682 16959 leader_election.cc:302] T 00000000000000000000000000000000 P 2bf14fb7b3b3456d910c3d46f4604b37 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2bf14fb7b3b3456d910c3d46f4604b37; no voters:
I20260619 18:49:58.250237 16959 leader_election.cc:288] T 00000000000000000000000000000000 P 2bf14fb7b3b3456d910c3d46f4604b37 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 18:49:58.250361 16963 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2bf14fb7b3b3456d910c3d46f4604b37 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 18:49:58.251446 16963 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2bf14fb7b3b3456d910c3d46f4604b37 [term 2 LEADER]: Becoming Leader. State: Replica: 2bf14fb7b3b3456d910c3d46f4604b37, State: Running, Role: LEADER
I20260619 18:49:58.252048 16963 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2bf14fb7b3b3456d910c3d46f4604b37 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bf14fb7b3b3456d910c3d46f4604b37" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 44203 } }
I20260619 18:49:58.252365 16959 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2bf14fb7b3b3456d910c3d46f4604b37 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:49:58.254381 16965 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bf14fb7b3b3456d910c3d46f4604b37 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2bf14fb7b3b3456d910c3d46f4604b37. Latest consensus state: current_term: 2 leader_uuid: "2bf14fb7b3b3456d910c3d46f4604b37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bf14fb7b3b3456d910c3d46f4604b37" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 44203 } } }
I20260619 18:49:58.254741 16965 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bf14fb7b3b3456d910c3d46f4604b37 [sys.catalog]: This master's current role is: LEADER
I20260619 18:49:58.255229 16964 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bf14fb7b3b3456d910c3d46f4604b37 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "2bf14fb7b3b3456d910c3d46f4604b37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bf14fb7b3b3456d910c3d46f4604b37" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 44203 } } }
I20260619 18:49:58.255491 16964 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bf14fb7b3b3456d910c3d46f4604b37 [sys.catalog]: This master's current role is: LEADER
I20260619 18:49:58.260496 16971 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:49:58.266185 16971 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:49:58.267678 16971 catalog_manager.cc:1259] Loaded cluster ID: d83e853089274f1282da7102ac311463
I20260619 18:49:58.267838 16971 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:49:58.271898 16971 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:49:58.273355 16971 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2bf14fb7b3b3456d910c3d46f4604b37: Loaded TSK: 0
I20260619 18:49:58.275015 16971 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:49:58.277458 16959 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260619 18:49:58.622280 16961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:49:58.622696 16961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:49:58.622885 16961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:49:58.634138 16961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:49:58.634332 16961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:49:58.634545 16961 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.65
I20260619 18:49:58.649730 16961 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:42157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.243.84:36633/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.65.keytab
--principal=kudu/127.3.243.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.65:39543
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.243.65
--webserver_port=45311
--tserver_master_addrs=127.3.243.126:44203
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.65
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:49:58.651654 16961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:49:58.653543 16961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:49:58.686044 16996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:49:58.687601 16997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:49:58.692186 16961 server_base.cc:1034] running on GCE node
W20260619 18:49:58.692860 16999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:49:58.693748 16961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:49:58.696787 16961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:49:58.702644 16961 hybrid_clock.cc:648] HybridClock initialized: now 1781894998700780 us; error 1849 us; skew 500 ppm
I20260619 18:49:58.703107 16961 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:49:58 dist-test-slave-m9wg krb5kdc[15913](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894998, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:49:58.707223 16961 init.cc:377] Logged in from keytab as kudu/127.3.243.65@KRBTEST.COM (short username kudu)
I20260619 18:49:58.709354 16961 webserver.cc:460] Webserver started at http://127.3.243.65:45311/ using document root <none> and password file <none>
I20260619 18:49:58.710299 16961 fs_manager.cc:359] Metadata directory not provided
I20260619 18:49:58.710490 16961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 18:49:58 dist-test-slave-m9wg krb5kdc[15913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894998, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.65@KRBTEST.COM for HTTP/127.3.243.84@KRBTEST.COM
I20260619 18:49:58.754894 16961 fs_manager.cc:709] Time spent opening directory manager: real 0.013s user 0.002s sys 0.007s
I20260619 18:49:58.768752 17007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:49:58.773593 16961 fs_manager.cc:725] Time spent opening block manager: real 0.017s user 0.009s sys 0.004s
I20260619 18:49:58.773885 16961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ts-0/wal
uuid: "903297c2967c48f0a501243b94829f0d"
format_stamp: "Formatted at 2026-06-19 18:49:55 on dist-test-slave-m9wg"
server_key: "110b065ea475454f77babd6b10a2c627"
server_key_iv: "067e8ae792b0c8a7f35d0775aefdc17a"
server_key_version: "kuduclusterkey@0"
I20260619 18:49:58.775493 16961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:49:58.817705 16961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:49:58.823083 16961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:49:58.823472 16961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:49:58.824813 16961 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:49:58.826706 16961 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:49:58.826845 16961 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:49:58.826967 16961 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:49:58.827068 16961 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:49:58.870666 16961 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.65:39543
I20260619 18:49:58.870738 17120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.65:39543 every 8 connection(s)
I20260619 18:49:58.872671 16961 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ts-0/data/info.pb
I20260619 18:49:58.873615 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 16961
I20260619 18:49:58.874396 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.243.84:36633/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.66:35753
--local_ip_for_outbound_sockets=127.3.243.66
--tserver_master_addrs=127.3.243.126:44203
--webserver_port=41495
--webserver_interface=127.3.243.66
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.66.keytab
--principal=kudu/127.3.243.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:42157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.243.84:36633/kms with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 18:49:58 dist-test-slave-m9wg krb5kdc[15913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894998, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.65@KRBTEST.COM for kudu/127.3.243.126@KRBTEST.COM
I20260619 18:49:58.918304 17124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:49:58.887401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:44203 (local address 127.3.243.65:54143)
0619 18:49:58.888846 (+ 1445us) negotiation.cc:107] Waiting for socket to connect
0619 18:49:58.889574 (+ 728us) client_negotiation.cc:174] Beginning negotiation
0619 18:49:58.892119 (+ 2545us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:49:58.898076 (+ 5957us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:49:58.898087 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:49:58.898818 (+ 731us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:49:58.899663 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:49:58.899690 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:49:58.902131 (+ 2441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:49:58.902137 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:49:58.904199 (+ 2062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:49:58.904215 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:49:58.905969 (+ 1754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:49:58.905976 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:49:58.906349 (+ 373us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:49:58.908350 (+ 2001us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:49:58.908389 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:49:58.910654 (+ 2265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:49:58.913717 (+ 3063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:49:58.913727 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:49:58.913744 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:49:58.914153 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:49:58.914591 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:49:58.914596 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:49:58.914600 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:49:58.914803 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:49:58.915784 (+ 981us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:49:58.915790 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:49:58.916109 (+ 319us) client_negotiation.cc:769] Sending connection context
0619 18:49:58.916394 (+ 285us) client_negotiation.cc:240] Negotiation successful
0619 18:49:58.916946 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1150,"thread_start_us":929,"threads_started":1}
I20260619 18:49:58.918304 17125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:49:58.888846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.65:54143 (local address 127.3.243.126:44203)
0619 18:49:58.896650 (+ 7804us) server_negotiation.cc:206] Beginning negotiation
0619 18:49:58.896669 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 18:49:58.896717 (+ 48us) server_negotiation.cc:407] Connection header received
0619 18:49:58.897084 (+ 367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:49:58.897100 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:49:58.897547 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:49:58.897914 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:49:58.899940 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:49:58.901917 (+ 1977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:49:58.904373 (+ 2456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:49:58.905418 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:49:58.905590 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:49:58.910902 (+ 5312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:49:58.910910 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:49:58.910927 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:49:58.910964 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:49:58.913580 (+ 2616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:49:58.914305 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:49:58.914312 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:49:58.914318 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:49:58.914471 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:49:58.914916 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:49:58.914919 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:49:58.914922 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:49:58.915504 (+ 582us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:49:58.915701 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0619 18:49:58.916650 (+ 949us) server_negotiation.cc:299] Negotiation successful
0619 18:49:58.917014 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7560,"thread_start_us":7349,"threads_started":1}
I20260619 18:49:58.922349 17121 heartbeater.cc:347] Connected to a master server at 127.3.243.126:44203
I20260619 18:49:58.922629 17121 heartbeater.cc:464] Registering TS with master...
I20260619 18:49:58.923307 17121 heartbeater.cc:511] Master 127.3.243.126:44203 requested a full tablet report, sending...
W20260619 18:49:58.925815 17121 heartbeater.cc:644] Failed to heartbeat to 127.3.243.126:44203 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 18:49:58.926024 17121 heartbeater.cc:464] Registering TS with master...
I20260619 18:49:58.926257 17121 heartbeater.cc:511] Master 127.3.243.126:44203 requested a full tablet report, sending...
I20260619 18:49:58.927249 17121 heartbeater.cc:464] Registering TS with master...
I20260619 18:49:58.927475 17121 heartbeater.cc:511] Master 127.3.243.126:44203 requested a full tablet report, sending...
W20260619 18:49:58.928409 17121 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 18:49:59.263562 17123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:49:59.263829 17123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:49:59.263944 17123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:49:59.274971 17123 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:49:59.275104 17123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:49:59.275269 17123 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.66
I20260619 18:49:59.288592 17123 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:42157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.243.84:36633/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.66.keytab
--principal=kudu/127.3.243.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.66:35753
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.243.66
--webserver_port=41495
--tserver_master_addrs=127.3.243.126:44203
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.66
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:49:59.289901 17123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:49:59.291549 17123 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:49:59.307153 17139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:49:59.307875 17140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:49:59.309508 17143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:49:59.309762 17123 server_base.cc:1034] running on GCE node
I20260619 18:49:59.310493 17123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:49:59.311626 17123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:49:59.312836 17123 hybrid_clock.cc:648] HybridClock initialized: now 1781894999312779 us; error 53 us; skew 500 ppm
I20260619 18:49:59.313165 17123 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:49:59 dist-test-slave-m9wg krb5kdc[15913](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894999, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:49:59.316219 17123 init.cc:377] Logged in from keytab as kudu/127.3.243.66@KRBTEST.COM (short username kudu)
I20260619 18:49:59.321693 17123 webserver.cc:460] Webserver started at http://127.3.243.66:41495/ using document root <none> and password file <none>
I20260619 18:49:59.322371 17123 fs_manager.cc:359] Metadata directory not provided
I20260619 18:49:59.322496 17123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 18:49:59 dist-test-slave-m9wg krb5kdc[15913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894999, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.66@KRBTEST.COM for HTTP/127.3.243.84@KRBTEST.COM
I20260619 18:49:59.369119 17123 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.004s sys 0.000s
I20260619 18:49:59.386880 17152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:49:59.389081 17123 fs_manager.cc:725] Time spent opening block manager: real 0.018s user 0.009s sys 0.008s
I20260619 18:49:59.389263 17123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ts-1/wal
uuid: "fb9191a13d5d4605aa972c67ee2c6f0d"
format_stamp: "Formatted at 2026-06-19 18:49:56 on dist-test-slave-m9wg"
server_key: "20a81610412617f4b5f9530c8d3171fe"
server_key_iv: "f03fff718b484eda37c17d3398308a46"
server_key_version: "kuduclusterkey@0"
I20260619 18:49:59.390743 17123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:49:59.414268 17123 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:49:59.419420 17123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:49:59.419770 17123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:49:59.421060 17123 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:49:59.426054 17123 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:49:59.426162 17123 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:49:59.426268 17123 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:49:59.426337 17123 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 18:49:59 dist-test-slave-m9wg krb5kdc[15913](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781894999, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:49:59.504257 17123 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.66:35753
I20260619 18:49:59.504330 17264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.66:35753 every 8 connection(s)
I20260619 18:49:59.505940 17123 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ts-1/data/info.pb
I20260619 18:49:59.512036 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 17123
I20260619 18:49:59.512820 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.243.84:36633/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.67:33993
--local_ip_for_outbound_sockets=127.3.243.67
--tserver_master_addrs=127.3.243.126:44203
--webserver_port=37917
--webserver_interface=127.3.243.67
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.67.keytab
--principal=kudu/127.3.243.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:42157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.243.84:36633/kms with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 18:49:59 dist-test-slave-m9wg krb5kdc[15913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894999, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.66@KRBTEST.COM for kudu/127.3.243.126@KRBTEST.COM
I20260619 18:49:59.529199 17268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:49:59.508573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.66:33109 (local address 127.3.243.126:44203)
0619 18:49:59.509024 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0619 18:49:59.509031 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:49:59.510106 (+ 1075us) server_negotiation.cc:407] Connection header received
0619 18:49:59.512092 (+ 1986us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:49:59.512099 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:49:59.512219 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:49:59.512300 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:49:59.514462 (+ 2162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:49:59.515495 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:49:59.517737 (+ 2242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:49:59.518279 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:49:59.518374 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:49:59.522503 (+ 4129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:49:59.522509 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:49:59.522514 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:49:59.522550 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:49:59.525731 (+ 3181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:49:59.526582 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:49:59.526588 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:49:59.526592 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:49:59.526731 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:49:59.527241 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:49:59.527246 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:49:59.527249 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:49:59.527608 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:49:59.527729 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0619 18:49:59.528661 (+ 932us) server_negotiation.cc:299] Negotiation successful
0619 18:49:59.528865 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":202,"threads_started":1}
I20260619 18:49:59.531342 17269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:49:59.509191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:44203 (local address 127.3.243.66:33109)
0619 18:49:59.509935 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0619 18:49:59.509998 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0619 18:49:59.511743 (+ 1745us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:49:59.512840 (+ 1097us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:49:59.512852 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:49:59.513474 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:49:59.514270 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:49:59.514295 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:49:59.515690 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:49:59.515694 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:49:59.517585 (+ 1891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:49:59.517600 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:49:59.518693 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:49:59.518698 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:49:59.518954 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:49:59.519710 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:49:59.519736 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:49:59.522281 (+ 2545us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:49:59.525888 (+ 3607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:49:59.525898 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:49:59.525916 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:49:59.526445 (+ 529us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:49:59.526865 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:49:59.526869 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:49:59.526872 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:49:59.527114 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:49:59.527832 (+ 718us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:49:59.527840 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:49:59.528273 (+ 433us) client_negotiation.cc:769] Sending connection context
0619 18:49:59.529506 (+ 1233us) client_negotiation.cc:240] Negotiation successful
0619 18:49:59.530038 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":291,"threads_started":1}
I20260619 18:49:59.533064 17266 heartbeater.cc:347] Connected to a master server at 127.3.243.126:44203
I20260619 18:49:59.533365 17266 heartbeater.cc:464] Registering TS with master...
I20260619 18:49:59.534175 17266 heartbeater.cc:511] Master 127.3.243.126:44203 requested a full tablet report, sending...
I20260619 18:49:59.536329 16924 ts_manager.cc:194] Registered new tserver with Master: fb9191a13d5d4605aa972c67ee2c6f0d (127.3.243.66:35753)
I20260619 18:49:59.538501 16924 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.66@KRBTEST.COM'} at 127.3.243.66:33109
Jun 19 18:49:59 dist-test-slave-m9wg krb5kdc[15913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894999, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.126@KRBTEST.COM for HTTP/127.3.243.84@KRBTEST.COM
I20260619 18:49:59.928566 17121 heartbeater.cc:464] Registering TS with master...
I20260619 18:49:59.929001 17121 heartbeater.cc:511] Master 127.3.243.126:44203 requested a full tablet report, sending...
I20260619 18:49:59.930358 16924 ts_manager.cc:194] Registered new tserver with Master: 903297c2967c48f0a501243b94829f0d (127.3.243.65:39543)
I20260619 18:49:59.931509 16924 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.65@KRBTEST.COM'} at 127.3.243.65:54143
W20260619 18:49:59.954236 17270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:49:59.954603 17270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:49:59.954783 17270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:49:59.965961 17270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:49:59.966102 17270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:49:59.966336 17270 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.67
I20260619 18:49:59.981364 17270 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:42157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.243.84:36633/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.67.keytab
--principal=kudu/127.3.243.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.67:33993
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.243.67
--webserver_port=37917
--tserver_master_addrs=127.3.243.126:44203
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.67
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:49:59.982936 17270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:49:59.984545 17270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:50:00.000386 17275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:50:00.000464 17276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:50:00.009660 17278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:50:00.010021 17270 server_base.cc:1034] running on GCE node
I20260619 18:50:00.010746 17270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:50:00.011891 17270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:50:00.013108 17270 hybrid_clock.cc:648] HybridClock initialized: now 1781895000013037 us; error 59 us; skew 500 ppm
I20260619 18:50:00.013466 17270 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:50:00 dist-test-slave-m9wg krb5kdc[15913](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895000, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:50:00.017375 17270 init.cc:377] Logged in from keytab as kudu/127.3.243.67@KRBTEST.COM (short username kudu)
I20260619 18:50:00.019176 17270 webserver.cc:460] Webserver started at http://127.3.243.67:37917/ using document root <none> and password file <none>
I20260619 18:50:00.020017 17270 fs_manager.cc:359] Metadata directory not provided
I20260619 18:50:00.020172 17270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 18:50:00 dist-test-slave-m9wg krb5kdc[15913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895000, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.67@KRBTEST.COM for HTTP/127.3.243.84@KRBTEST.COM
I20260619 18:50:00.055194 17270 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 18:50:00.058947 17286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:50:00.060787 17270 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 18:50:00.061014 17270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ts-2/wal
uuid: "0f9b804b0e4d4d98978b08e172a9d104"
format_stamp: "Formatted at 2026-06-19 18:49:57 on dist-test-slave-m9wg"
server_key: "050b36ea704407cfae3570c5a851c7c0"
server_key_iv: "40b3f740ac975a0e57699fb3429423a4"
server_key_version: "kuduclusterkey@0"
I20260619 18:50:00.062382 17270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Jun 19 18:50:00 dist-test-slave-m9wg krb5kdc[15913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894999, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.126@KRBTEST.COM for HTTP/127.3.243.84@KRBTEST.COM
I20260619 18:50:00.124357 17270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:50:00.129459 17270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:50:00.129843 17270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:50:00.139994 17270 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:50:00.148681 17270 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:50:00.148874 17270 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260619 18:50:00.149092 17270 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:50:00.149200 17270 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:50:00.309667 17270 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.67:33993
I20260619 18:50:00.309767 17400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.67:33993 every 8 connection(s)
I20260619 18:50:00.311746 17270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ts-2/data/info.pb
I20260619 18:50:00.320128 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 17270
Jun 19 18:50:00 dist-test-slave-m9wg krb5kdc[15913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895000, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.67@KRBTEST.COM for kudu/127.3.243.126@KRBTEST.COM
I20260619 18:50:00.351529 17408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:00.321248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43650 (local address 127.3.243.126:44203)
0619 18:50:00.336567 (+ 15319us) server_negotiation.cc:206] Beginning negotiation
0619 18:50:00.336573 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:50:00.336592 (+ 19us) server_negotiation.cc:407] Connection header received
0619 18:50:00.336655 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:50:00.336660 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:50:00.336734 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:50:00.336846 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:50:00.338738 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:00.339954 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:00.341306 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:00.341724 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:00.341837 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:00.344885 (+ 3048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:50:00.344891 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:50:00.344896 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:50:00.344932 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:50:00.348719 (+ 3787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:00.349423 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:00.349429 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:00.349433 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:00.349591 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:00.350018 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:00.350023 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:00.350026 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:00.350344 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:50:00.350599 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0619 18:50:00.350924 (+ 325us) server_negotiation.cc:299] Negotiation successful
0619 18:50:00.351141 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15203,"thread_start_us":15099,"threads_started":1}
I20260619 18:50:00.353583 17405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:00.316431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.67:52427 (local address 127.3.243.126:44203)
0619 18:50:00.335953 (+ 19522us) server_negotiation.cc:206] Beginning negotiation
0619 18:50:00.335957 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:50:00.335973 (+ 16us) server_negotiation.cc:407] Connection header received
0619 18:50:00.336038 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:50:00.336042 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:50:00.336101 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:50:00.336248 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:50:00.338331 (+ 2083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:00.339450 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:00.341976 (+ 2526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:00.342616 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:00.342732 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:00.346350 (+ 3618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:50:00.346355 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:50:00.346358 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:50:00.346400 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:50:00.349927 (+ 3527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:00.350681 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:00.350686 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:00.350689 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:00.350828 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:00.351364 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:00.351368 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:00.351371 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:00.351687 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:50:00.351818 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 18:50:00.352746 (+ 928us) server_negotiation.cc:299] Negotiation successful
0619 18:50:00.353152 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19422,"thread_start_us":19313,"threads_started":1}
I20260619 18:50:00.354524 17404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:00.315845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:44203 (local address 127.3.243.67:52427)
0619 18:50:00.332723 (+ 16878us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:00.332782 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 18:50:00.335253 (+ 2471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:50:00.336703 (+ 1450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:50:00.336711 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:50:00.337300 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:50:00.338121 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:50:00.338157 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:50:00.339644 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:50:00.339649 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:50:00.341804 (+ 2155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:50:00.341820 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:50:00.342843 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:50:00.342851 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:50:00.343120 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:00.343823 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:50:00.343857 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:50:00.346107 (+ 2250us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:50:00.350077 (+ 3970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:50:00.350084 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:50:00.350096 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:50:00.350528 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:50:00.350950 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:50:00.350954 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:50:00.350957 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:50:00.351225 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:50:00.351885 (+ 660us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:50:00.351890 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:50:00.352329 (+ 439us) client_negotiation.cc:769] Sending connection context
0619 18:50:00.352648 (+ 319us) client_negotiation.cc:240] Negotiation successful
0619 18:50:00.353184 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16579,"thread_start_us":16369,"threads_started":1}
I20260619 18:50:00.356004 17402 heartbeater.cc:347] Connected to a master server at 127.3.243.126:44203
I20260619 18:50:00.356343 17402 heartbeater.cc:464] Registering TS with master...
I20260619 18:50:00.357018 17402 heartbeater.cc:511] Master 127.3.243.126:44203 requested a full tablet report, sending...
I20260619 18:50:00.358407 16924 ts_manager.cc:194] Registered new tserver with Master: 0f9b804b0e4d4d98978b08e172a9d104 (127.3.243.67:33993)
I20260619 18:50:00.359509 16924 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.67@KRBTEST.COM'} at 127.3.243.67:52427
I20260619 18:50:00.366482 4045 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 18:50:00.383746 17405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:00.370078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43654 (local address 127.3.243.126:44203)
0619 18:50:00.370257 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 18:50:00.370263 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:50:00.370471 (+ 208us) server_negotiation.cc:407] Connection header received
0619 18:50:00.370610 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:50:00.370616 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:50:00.370700 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:50:00.370806 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:50:00.372301 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:00.373343 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:00.374675 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:00.375337 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:00.375460 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:00.377840 (+ 2380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:50:00.377847 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:50:00.377852 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:50:00.377890 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:50:00.381263 (+ 3373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:00.381954 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:00.381959 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:00.381963 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:00.382121 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:00.382423 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:00.382427 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:00.382431 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:00.382771 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:50:00.383035 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0619 18:50:00.383204 (+ 169us) server_negotiation.cc:299] Negotiation successful
0619 18:50:00.383419 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 18:50:00.388442 16924 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43654:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:50:00.395761 16924 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:50:00.433473 17424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:00.410037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32836 (local address 127.3.243.66:35753)
0619 18:50:00.423865 (+ 13828us) server_negotiation.cc:206] Beginning negotiation
0619 18:50:00.423874 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:50:00.423906 (+ 32us) server_negotiation.cc:407] Connection header received
0619 18:50:00.424021 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:50:00.424029 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:50:00.424335 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:50:00.424549 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:50:00.425672 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:00.427273 (+ 1601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:00.430107 (+ 2834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:00.431606 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:00.431729 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:00.431925 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0619 18:50:00.432999 (+ 1074us) server_negotiation.cc:299] Negotiation successful
0619 18:50:00.433179 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13650,"thread_start_us":14689,"threads_started":1}
I20260619 18:50:00.434949 17422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:00.409869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.66:35753 (local address 127.0.0.1:32836)
0619 18:50:00.422068 (+ 12199us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:00.422111 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 18:50:00.422304 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:50:00.425008 (+ 2704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:50:00.425015 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:50:00.425048 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:50:00.425454 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:50:00.425469 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:50:00.427434 (+ 1965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:50:00.427440 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:50:00.429898 (+ 2458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:50:00.429914 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:50:00.432084 (+ 2170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:50:00.432090 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:50:00.432317 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:00.432377 (+ 60us) client_negotiation.cc:769] Sending connection context
0619 18:50:00.432524 (+ 147us) client_negotiation.cc:240] Negotiation successful
0619 18:50:00.432742 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11896,"thread_start_us":12602,"threads_started":1}
I20260619 18:50:00.437872 17426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:00.420251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.65:39543 (local address 127.0.0.1:50904)
0619 18:50:00.429090 (+ 8839us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:00.429106 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 18:50:00.429213 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:50:00.429972 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:50:00.429977 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:50:00.429999 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:50:00.430336 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:50:00.430347 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:50:00.432387 (+ 2040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:50:00.432392 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:50:00.434908 (+ 2516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:50:00.434924 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:50:00.437068 (+ 2144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:50:00.437075 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:50:00.437260 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:00.437299 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 18:50:00.437373 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 18:50:00.437502 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8716,"mutex_wait_us":4,"thread_start_us":8551,"threads_started":1}
I20260619 18:50:00.439002 17427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:00.420947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50904 (local address 127.3.243.65:39543)
0619 18:50:00.428314 (+ 7367us) server_negotiation.cc:206] Beginning negotiation
0619 18:50:00.428324 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:50:00.429455 (+ 1131us) server_negotiation.cc:407] Connection header received
0619 18:50:00.429565 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:50:00.429573 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:50:00.429803 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:50:00.429965 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:50:00.430528 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:00.432230 (+ 1702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:00.435251 (+ 3021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:00.436790 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:00.438210 (+ 1420us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:00.438439 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0619 18:50:00.438512 (+ 73us) server_negotiation.cc:299] Negotiation successful
0619 18:50:00.438667 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7185,"thread_start_us":7041,"threads_started":1}
I20260619 18:50:00.441301 17423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:00.410006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.67:33993 (local address 127.0.0.1:46826)
0619 18:50:00.423281 (+ 13275us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:00.423301 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 18:50:00.423411 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:50:00.433511 (+ 10100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:50:00.433516 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:50:00.433542 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:50:00.434149 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:50:00.434162 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:50:00.436523 (+ 2361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:50:00.436528 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:50:00.438615 (+ 2087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:50:00.438630 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:50:00.440554 (+ 1924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:50:00.440560 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:50:00.440737 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:00.440768 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 18:50:00.440836 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 18:50:00.440952 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13156,"mutex_wait_us":1,"thread_start_us":14200,"threads_started":1}
I20260619 18:50:00.442008 17425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:00.410677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46826 (local address 127.3.243.67:33993)
0619 18:50:00.432447 (+ 21770us) server_negotiation.cc:206] Beginning negotiation
0619 18:50:00.432456 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:50:00.432490 (+ 34us) server_negotiation.cc:407] Connection header received
0619 18:50:00.432582 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:50:00.432589 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:50:00.433283 (+ 694us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:50:00.434349 (+ 1066us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:50:00.434614 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:00.436379 (+ 1765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:00.438805 (+ 2426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:00.440335 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:00.441190 (+ 855us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:00.441411 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0619 18:50:00.441476 (+ 65us) server_negotiation.cc:299] Negotiation successful
0619 18:50:00.441627 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21588,"thread_start_us":22189,"threads_started":1}
I20260619 18:50:00.439038 17200 tablet_service.cc:1467] Processing CreateTablet for tablet 83a92a069f7d400c91db23e4444469fc (DEFAULT_TABLE table=test-table [id=c9e7c345891b4cbabcd62c0ea6217d8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:50:00.443257 17056 tablet_service.cc:1467] Processing CreateTablet for tablet 83a92a069f7d400c91db23e4444469fc (DEFAULT_TABLE table=test-table [id=c9e7c345891b4cbabcd62c0ea6217d8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:50:00.444418 17200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83a92a069f7d400c91db23e4444469fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:50:00.444633 17056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83a92a069f7d400c91db23e4444469fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:50:00.445333 17335 tablet_service.cc:1467] Processing CreateTablet for tablet 83a92a069f7d400c91db23e4444469fc (DEFAULT_TABLE table=test-table [id=c9e7c345891b4cbabcd62c0ea6217d8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:50:00.446592 17335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83a92a069f7d400c91db23e4444469fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:50:00.465348 17430 tablet_bootstrap.cc:492] T 83a92a069f7d400c91db23e4444469fc P 0f9b804b0e4d4d98978b08e172a9d104: Bootstrap starting.
I20260619 18:50:00.468093 17428 tablet_bootstrap.cc:492] T 83a92a069f7d400c91db23e4444469fc P 903297c2967c48f0a501243b94829f0d: Bootstrap starting.
I20260619 18:50:00.469239 17429 tablet_bootstrap.cc:492] T 83a92a069f7d400c91db23e4444469fc P fb9191a13d5d4605aa972c67ee2c6f0d: Bootstrap starting.
I20260619 18:50:00.471952 17430 tablet_bootstrap.cc:654] T 83a92a069f7d400c91db23e4444469fc P 0f9b804b0e4d4d98978b08e172a9d104: Neither blocks nor log segments found. Creating new log.
I20260619 18:50:00.472651 17429 tablet_bootstrap.cc:654] T 83a92a069f7d400c91db23e4444469fc P fb9191a13d5d4605aa972c67ee2c6f0d: Neither blocks nor log segments found. Creating new log.
I20260619 18:50:00.472792 17428 tablet_bootstrap.cc:654] T 83a92a069f7d400c91db23e4444469fc P 903297c2967c48f0a501243b94829f0d: Neither blocks nor log segments found. Creating new log.
I20260619 18:50:00.473270 17430 log.cc:824] T 83a92a069f7d400c91db23e4444469fc P 0f9b804b0e4d4d98978b08e172a9d104: Log is configured to *not* fsync() on all Append() calls
I20260619 18:50:00.473899 17429 log.cc:824] T 83a92a069f7d400c91db23e4444469fc P fb9191a13d5d4605aa972c67ee2c6f0d: Log is configured to *not* fsync() on all Append() calls
I20260619 18:50:00.473981 17428 log.cc:824] T 83a92a069f7d400c91db23e4444469fc P 903297c2967c48f0a501243b94829f0d: Log is configured to *not* fsync() on all Append() calls
I20260619 18:50:00.484756 17428 tablet_bootstrap.cc:492] T 83a92a069f7d400c91db23e4444469fc P 903297c2967c48f0a501243b94829f0d: No bootstrap required, opened a new log
I20260619 18:50:00.484759 17429 tablet_bootstrap.cc:492] T 83a92a069f7d400c91db23e4444469fc P fb9191a13d5d4605aa972c67ee2c6f0d: No bootstrap required, opened a new log
I20260619 18:50:00.485049 17429 ts_tablet_manager.cc:1397] T 83a92a069f7d400c91db23e4444469fc P fb9191a13d5d4605aa972c67ee2c6f0d: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.004s
I20260619 18:50:00.485038 17428 ts_tablet_manager.cc:1397] T 83a92a069f7d400c91db23e4444469fc P 903297c2967c48f0a501243b94829f0d: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.008s
I20260619 18:50:00.486811 17430 tablet_bootstrap.cc:492] T 83a92a069f7d400c91db23e4444469fc P 0f9b804b0e4d4d98978b08e172a9d104: No bootstrap required, opened a new log
I20260619 18:50:00.487177 17430 ts_tablet_manager.cc:1397] T 83a92a069f7d400c91db23e4444469fc P 0f9b804b0e4d4d98978b08e172a9d104: Time spent bootstrapping tablet: real 0.022s user 0.009s sys 0.009s
I20260619 18:50:00.495718 17428 raft_consensus.cc:348] T 83a92a069f7d400c91db23e4444469fc P 903297c2967c48f0a501243b94829f0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb9191a13d5d4605aa972c67ee2c6f0d" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 35753 } } peers { permanent_uuid: "0f9b804b0e4d4d98978b08e172a9d104" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 33993 } } peers { permanent_uuid: "903297c2967c48f0a501243b94829f0d" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 39543 } }
I20260619 18:50:00.495718 17429 raft_consensus.cc:348] T 83a92a069f7d400c91db23e4444469fc P fb9191a13d5d4605aa972c67ee2c6f0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb9191a13d5d4605aa972c67ee2c6f0d" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 35753 } } peers { permanent_uuid: "0f9b804b0e4d4d98978b08e172a9d104" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 33993 } } peers { permanent_uuid: "903297c2967c48f0a501243b94829f0d" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 39543 } }
I20260619 18:50:00.496194 17429 raft_consensus.cc:374] T 83a92a069f7d400c91db23e4444469fc P fb9191a13d5d4605aa972c67ee2c6f0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:50:00.496212 17428 raft_consensus.cc:374] T 83a92a069f7d400c91db23e4444469fc P 903297c2967c48f0a501243b94829f0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:50:00.496308 17428 raft_consensus.cc:729] T 83a92a069f7d400c91db23e4444469fc P 903297c2967c48f0a501243b94829f0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 903297c2967c48f0a501243b94829f0d, State: Initialized, Role: FOLLOWER
I20260619 18:50:00.496309 17429 raft_consensus.cc:729] T 83a92a069f7d400c91db23e4444469fc P fb9191a13d5d4605aa972c67ee2c6f0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb9191a13d5d4605aa972c67ee2c6f0d, State: Initialized, Role: FOLLOWER
I20260619 18:50:00.496965 17428 consensus_queue.cc:260] T 83a92a069f7d400c91db23e4444469fc P 903297c2967c48f0a501243b94829f0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb9191a13d5d4605aa972c67ee2c6f0d" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 35753 } } peers { permanent_uuid: "0f9b804b0e4d4d98978b08e172a9d104" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 33993 } } peers { permanent_uuid: "903297c2967c48f0a501243b94829f0d" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 39543 } }
I20260619 18:50:00.496971 17429 consensus_queue.cc:260] T 83a92a069f7d400c91db23e4444469fc P fb9191a13d5d4605aa972c67ee2c6f0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb9191a13d5d4605aa972c67ee2c6f0d" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 35753 } } peers { permanent_uuid: "0f9b804b0e4d4d98978b08e172a9d104" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 33993 } } peers { permanent_uuid: "903297c2967c48f0a501243b94829f0d" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 39543 } }
I20260619 18:50:00.496953 17430 raft_consensus.cc:348] T 83a92a069f7d400c91db23e4444469fc P 0f9b804b0e4d4d98978b08e172a9d104 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb9191a13d5d4605aa972c67ee2c6f0d" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 35753 } } peers { permanent_uuid: "0f9b804b0e4d4d98978b08e172a9d104" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 33993 } } peers { permanent_uuid: "903297c2967c48f0a501243b94829f0d" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 39543 } }
I20260619 18:50:00.497314 17430 raft_consensus.cc:374] T 83a92a069f7d400c91db23e4444469fc P 0f9b804b0e4d4d98978b08e172a9d104 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:50:00.497399 17430 raft_consensus.cc:729] T 83a92a069f7d400c91db23e4444469fc P 0f9b804b0e4d4d98978b08e172a9d104 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f9b804b0e4d4d98978b08e172a9d104, State: Initialized, Role: FOLLOWER
I20260619 18:50:00.498119 17266 heartbeater.cc:503] Master 127.3.243.126:44203 was elected leader, sending a full tablet report...
I20260619 18:50:00.498075 17430 consensus_queue.cc:260] T 83a92a069f7d400c91db23e4444469fc P 0f9b804b0e4d4d98978b08e172a9d104 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb9191a13d5d4605aa972c67ee2c6f0d" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 35753 } } peers { permanent_uuid: "0f9b804b0e4d4d98978b08e172a9d104" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 33993 } } peers { permanent_uuid: "903297c2967c48f0a501243b94829f0d" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 39543 } }
I20260619 18:50:00.498834 17429 ts_tablet_manager.cc:1428] T 83a92a069f7d400c91db23e4444469fc P fb9191a13d5d4605aa972c67ee2c6f0d: Time spent starting tablet: real 0.014s user 0.009s sys 0.003s
I20260619 18:50:00.499274 17121 heartbeater.cc:503] Master 127.3.243.126:44203 was elected leader, sending a full tablet report...
I20260619 18:50:00.501214 17402 heartbeater.cc:503] Master 127.3.243.126:44203 was elected leader, sending a full tablet report...
I20260619 18:50:00.501890 17430 ts_tablet_manager.cc:1428] T 83a92a069f7d400c91db23e4444469fc P 0f9b804b0e4d4d98978b08e172a9d104: Time spent starting tablet: real 0.015s user 0.010s sys 0.003s
I20260619 18:50:00.504181 17428 ts_tablet_manager.cc:1428] T 83a92a069f7d400c91db23e4444469fc P 903297c2967c48f0a501243b94829f0d: Time spent starting tablet: real 0.019s user 0.008s sys 0.006s
W20260619 18:50:00.508397 17267 tablet.cc:2369] T 83a92a069f7d400c91db23e4444469fc P fb9191a13d5d4605aa972c67ee2c6f0d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:50:00.519110 17436 raft_consensus.cc:482] T 83a92a069f7d400c91db23e4444469fc P 0f9b804b0e4d4d98978b08e172a9d104 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:50:00.519320 17436 raft_consensus.cc:504] T 83a92a069f7d400c91db23e4444469fc P 0f9b804b0e4d4d98978b08e172a9d104 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb9191a13d5d4605aa972c67ee2c6f0d" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 35753 } } peers { permanent_uuid: "0f9b804b0e4d4d98978b08e172a9d104" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 33993 } } peers { permanent_uuid: "903297c2967c48f0a501243b94829f0d" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 39543 } }
I20260619 18:50:00.520859 17436 leader_election.cc:288] T 83a92a069f7d400c91db23e4444469fc P 0f9b804b0e4d4d98978b08e172a9d104 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fb9191a13d5d4605aa972c67ee2c6f0d (127.3.243.66:35753), 903297c2967c48f0a501243b94829f0d (127.3.243.65:39543)
I20260619 18:50:00.525983 17427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:00.521313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.67:50857 (local address 127.3.243.65:39543)
0619 18:50:00.521544 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0619 18:50:00.521550 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:50:00.521568 (+ 18us) server_negotiation.cc:407] Connection header received
0619 18:50:00.521635 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:50:00.521639 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:50:00.521691 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:50:00.521759 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:50:00.522143 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:00.522823 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:00.524448 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:00.525196 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:00.525331 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:00.525400 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 18:50:00.525638 (+ 238us) server_negotiation.cc:299] Negotiation successful
0619 18:50:00.525760 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 18:50:00.525983 17404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:00.521172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.65:39543 (local address 127.3.243.67:50857)
0619 18:50:00.521359 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:00.521379 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 18:50:00.521496 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:50:00.521801 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:50:00.521806 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:50:00.521830 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:50:00.522015 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:50:00.522023 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:50:00.522944 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:50:00.522947 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:50:00.524284 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:50:00.524295 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:50:00.525350 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:50:00.525355 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:50:00.525504 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:00.525526 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 18:50:00.525604 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 18:50:00.525735 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69,"mutex_wait_us":1}
I20260619 18:50:00.526804 17424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:00.521170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.67:40127 (local address 127.3.243.66:35753)
0619 18:50:00.521388 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0619 18:50:00.521394 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:50:00.522178 (+ 784us) server_negotiation.cc:407] Connection header received
0619 18:50:00.522332 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:50:00.522336 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:50:00.522383 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:50:00.522452 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:50:00.522929 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:00.523606 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:00.525223 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:00.525969 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:00.526115 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:00.526181 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 18:50:00.526544 (+ 363us) server_negotiation.cc:299] Negotiation successful
0619 18:50:00.526639 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260619 18:50:00.526959 17437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:00.521230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.66:35753 (local address 127.3.243.67:40127)
0619 18:50:00.522088 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:00.522102 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 18:50:00.522212 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:50:00.522476 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:50:00.522480 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:50:00.522511 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:50:00.522787 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:50:00.522795 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:50:00.523742 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:50:00.523746 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:50:00.525076 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:50:00.525087 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:50:00.526281 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:50:00.526285 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:50:00.526394 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:00.526416 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 18:50:00.526496 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 18:50:00.526618 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":754,"mutex_wait_us":170,"thread_start_us":360,"threads_started":1}
I20260619 18:50:00.526948 17076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83a92a069f7d400c91db23e4444469fc" candidate_uuid: "0f9b804b0e4d4d98978b08e172a9d104" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "903297c2967c48f0a501243b94829f0d" is_pre_election: true
I20260619 18:50:00.527446 17076 raft_consensus.cc:2454] T 83a92a069f7d400c91db23e4444469fc P 903297c2967c48f0a501243b94829f0d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f9b804b0e4d4d98978b08e172a9d104 in term 0.
I20260619 18:50:00.527714 17220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83a92a069f7d400c91db23e4444469fc" candidate_uuid: "0f9b804b0e4d4d98978b08e172a9d104" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb9191a13d5d4605aa972c67ee2c6f0d" is_pre_election: true
I20260619 18:50:00.528024 17290 leader_election.cc:302] T 83a92a069f7d400c91db23e4444469fc P 0f9b804b0e4d4d98978b08e172a9d104 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f9b804b0e4d4d98978b08e172a9d104, 903297c2967c48f0a501243b94829f0d; no voters:
I20260619 18:50:00.528223 17220 raft_consensus.cc:2454] T 83a92a069f7d400c91db23e4444469fc P fb9191a13d5d4605aa972c67ee2c6f0d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f9b804b0e4d4d98978b08e172a9d104 in term 0.
I20260619 18:50:00.528438 17436 raft_consensus.cc:2781] T 83a92a069f7d400c91db23e4444469fc P 0f9b804b0e4d4d98978b08e172a9d104 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:50:00.528554 17436 raft_consensus.cc:482] T 83a92a069f7d400c91db23e4444469fc P 0f9b804b0e4d4d98978b08e172a9d104 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:50:00.528658 17436 raft_consensus.cc:3037] T 83a92a069f7d400c91db23e4444469fc P 0f9b804b0e4d4d98978b08e172a9d104 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:50:00.530704 17436 raft_consensus.cc:504] T 83a92a069f7d400c91db23e4444469fc P 0f9b804b0e4d4d98978b08e172a9d104 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb9191a13d5d4605aa972c67ee2c6f0d" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 35753 } } peers { permanent_uuid: "0f9b804b0e4d4d98978b08e172a9d104" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 33993 } } peers { permanent_uuid: "903297c2967c48f0a501243b94829f0d" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 39543 } }
I20260619 18:50:00.531204 17436 leader_election.cc:288] T 83a92a069f7d400c91db23e4444469fc P 0f9b804b0e4d4d98978b08e172a9d104 [CANDIDATE]: Term 1 election: Requested vote from peers fb9191a13d5d4605aa972c67ee2c6f0d (127.3.243.66:35753), 903297c2967c48f0a501243b94829f0d (127.3.243.65:39543)
I20260619 18:50:00.531585 17076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83a92a069f7d400c91db23e4444469fc" candidate_uuid: "0f9b804b0e4d4d98978b08e172a9d104" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "903297c2967c48f0a501243b94829f0d"
I20260619 18:50:00.531595 17220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83a92a069f7d400c91db23e4444469fc" candidate_uuid: "0f9b804b0e4d4d98978b08e172a9d104" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb9191a13d5d4605aa972c67ee2c6f0d"
I20260619 18:50:00.531735 17220 raft_consensus.cc:3037] T 83a92a069f7d400c91db23e4444469fc P fb9191a13d5d4605aa972c67ee2c6f0d [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:50:00.531746 17076 raft_consensus.cc:3037] T 83a92a069f7d400c91db23e4444469fc P 903297c2967c48f0a501243b94829f0d [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:50:00.533757 17076 raft_consensus.cc:2454] T 83a92a069f7d400c91db23e4444469fc P 903297c2967c48f0a501243b94829f0d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f9b804b0e4d4d98978b08e172a9d104 in term 1.
I20260619 18:50:00.533793 17220 raft_consensus.cc:2454] T 83a92a069f7d400c91db23e4444469fc P fb9191a13d5d4605aa972c67ee2c6f0d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f9b804b0e4d4d98978b08e172a9d104 in term 1.
I20260619 18:50:00.534093 17290 leader_election.cc:302] T 83a92a069f7d400c91db23e4444469fc P 0f9b804b0e4d4d98978b08e172a9d104 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f9b804b0e4d4d98978b08e172a9d104, 903297c2967c48f0a501243b94829f0d; no voters:
I20260619 18:50:00.534368 17436 raft_consensus.cc:2781] T 83a92a069f7d400c91db23e4444469fc P 0f9b804b0e4d4d98978b08e172a9d104 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:50:00.534803 17436 raft_consensus.cc:686] T 83a92a069f7d400c91db23e4444469fc P 0f9b804b0e4d4d98978b08e172a9d104 [term 1 LEADER]: Becoming Leader. State: Replica: 0f9b804b0e4d4d98978b08e172a9d104, State: Running, Role: LEADER
I20260619 18:50:00.535224 17436 consensus_queue.cc:237] T 83a92a069f7d400c91db23e4444469fc P 0f9b804b0e4d4d98978b08e172a9d104 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb9191a13d5d4605aa972c67ee2c6f0d" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 35753 } } peers { permanent_uuid: "0f9b804b0e4d4d98978b08e172a9d104" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 33993 } } peers { permanent_uuid: "903297c2967c48f0a501243b94829f0d" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 39543 } }
I20260619 18:50:00.539731 16923 catalog_manager.cc:5495] T 83a92a069f7d400c91db23e4444469fc P 0f9b804b0e4d4d98978b08e172a9d104 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0f9b804b0e4d4d98978b08e172a9d104 (127.3.243.67). New cstate: current_term: 1 leader_uuid: "0f9b804b0e4d4d98978b08e172a9d104" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb9191a13d5d4605aa972c67ee2c6f0d" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 35753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f9b804b0e4d4d98978b08e172a9d104" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 33993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "903297c2967c48f0a501243b94829f0d" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 39543 } health_report { overall_health: UNKNOWN } } }
W20260619 18:50:00.569381 17403 tablet.cc:2369] T 83a92a069f7d400c91db23e4444469fc P 0f9b804b0e4d4d98978b08e172a9d104: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:50:00.580456 17425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:00.575315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46832 (local address 127.3.243.67:33993)
0619 18:50:00.575491 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 18:50:00.575497 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:50:00.575533 (+ 36us) server_negotiation.cc:407] Connection header received
0619 18:50:00.575738 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:50:00.575741 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:50:00.575824 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:50:00.575943 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:50:00.576372 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:00.577068 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:00.578388 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:00.578788 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:00.578893 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:00.579378 (+ 485us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:50:00.579864 (+ 486us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:50:00.579992 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 18:50:00.580128 (+ 136us) server_negotiation.cc:299] Negotiation successful
0619 18:50:00.580231 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 18:50:00.587987 17220 raft_consensus.cc:1261] T 83a92a069f7d400c91db23e4444469fc P fb9191a13d5d4605aa972c67ee2c6f0d [term 1 FOLLOWER]: Refusing update from remote peer 0f9b804b0e4d4d98978b08e172a9d104: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:50:00.587975 17076 raft_consensus.cc:1261] T 83a92a069f7d400c91db23e4444469fc P 903297c2967c48f0a501243b94829f0d [term 1 FOLLOWER]: Refusing update from remote peer 0f9b804b0e4d4d98978b08e172a9d104: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:50:00.588968 17438 consensus_queue.cc:1035] T 83a92a069f7d400c91db23e4444469fc P 0f9b804b0e4d4d98978b08e172a9d104 [LEADER]: Connected to new peer: Peer: permanent_uuid: "903297c2967c48f0a501243b94829f0d" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 39543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:50:00.589286 17436 consensus_queue.cc:1035] T 83a92a069f7d400c91db23e4444469fc P 0f9b804b0e4d4d98978b08e172a9d104 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb9191a13d5d4605aa972c67ee2c6f0d" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 35753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:50:00.600651 17441 mvcc.cc:205] Tried to move back new op lower bound from 7298641922399318016 to 7298641922195935232. Current Snapshot: MvccSnapshot[applied={T|T < 7298641922399318016}]
I20260619 18:50:00.603169 17442 mvcc.cc:205] Tried to move back new op lower bound from 7298641922399318016 to 7298641922195935232. Current Snapshot: MvccSnapshot[applied={T|T < 7298641922399318016}]
I20260619 18:50:00.612274 16923 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43654:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:50:00.612566 16923 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43654:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:50:00.616087 16923 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2bf14fb7b3b3456d910c3d46f4604b37: Sending DeleteTablet for 3 replicas of tablet 83a92a069f7d400c91db23e4444469fc
I20260619 18:50:00.617210 17335 tablet_service.cc:1514] Processing DeleteTablet for tablet 83a92a069f7d400c91db23e4444469fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:50:00 UTC) from {username='kudu', principal='kudu/127.3.243.126@KRBTEST.COM'} at 127.0.0.1:46826
I20260619 18:50:00.617201 17200 tablet_service.cc:1514] Processing DeleteTablet for tablet 83a92a069f7d400c91db23e4444469fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:50:00 UTC) from {username='kudu', principal='kudu/127.3.243.126@KRBTEST.COM'} at 127.0.0.1:32836
I20260619 18:50:00.617268 17056 tablet_service.cc:1514] Processing DeleteTablet for tablet 83a92a069f7d400c91db23e4444469fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:50:00 UTC) from {username='kudu', principal='kudu/127.3.243.126@KRBTEST.COM'} at 127.0.0.1:50904
I20260619 18:50:00.620906 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 16961
I20260619 18:50:00.629560 17451 tablet_replica.cc:330] T 83a92a069f7d400c91db23e4444469fc P fb9191a13d5d4605aa972c67ee2c6f0d: stopping tablet replica
I20260619 18:50:00.630155 17450 tablet_replica.cc:330] T 83a92a069f7d400c91db23e4444469fc P 0f9b804b0e4d4d98978b08e172a9d104: stopping tablet replica
I20260619 18:50:00.630718 17450 raft_consensus.cc:2229] T 83a92a069f7d400c91db23e4444469fc P 0f9b804b0e4d4d98978b08e172a9d104 [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:50:00.631263 17451 raft_consensus.cc:2229] T 83a92a069f7d400c91db23e4444469fc P fb9191a13d5d4605aa972c67ee2c6f0d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:50:00.631839 17451 raft_consensus.cc:2258] T 83a92a069f7d400c91db23e4444469fc P fb9191a13d5d4605aa972c67ee2c6f0d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:50:00.632579 17450 raft_consensus.cc:2258] T 83a92a069f7d400c91db23e4444469fc P 0f9b804b0e4d4d98978b08e172a9d104 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:50:00.638093 17452 tablet_replica.cc:330] T 83a92a069f7d400c91db23e4444469fc P 903297c2967c48f0a501243b94829f0d: stopping tablet replica
I20260619 18:50:00.638502 17452 raft_consensus.cc:2229] T 83a92a069f7d400c91db23e4444469fc P 903297c2967c48f0a501243b94829f0d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:50:00.638998 17452 raft_consensus.cc:2258] T 83a92a069f7d400c91db23e4444469fc P 903297c2967c48f0a501243b94829f0d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:50:00.641163 17450 ts_tablet_manager.cc:1905] T 83a92a069f7d400c91db23e4444469fc P 0f9b804b0e4d4d98978b08e172a9d104: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:50:00.642513 17451 ts_tablet_manager.cc:1905] T 83a92a069f7d400c91db23e4444469fc P fb9191a13d5d4605aa972c67ee2c6f0d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:50:00.644596 17452 ts_tablet_manager.cc:1905] T 83a92a069f7d400c91db23e4444469fc P 903297c2967c48f0a501243b94829f0d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:50:00.647123 17427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:00.622249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50916 (local address 127.3.243.65:39543)
0619 18:50:00.622411 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 18:50:00.622417 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:50:00.622434 (+ 17us) server_negotiation.cc:407] Connection header received
0619 18:50:00.622497 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:50:00.622503 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:50:00.622575 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:50:00.622666 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:50:00.628080 (+ 5414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:00.629239 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:00.631293 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:00.631972 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:00.632091 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:00.635561 (+ 3470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:50:00.635577 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:50:00.635595 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:50:00.635642 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:50:00.642942 (+ 7300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:00.643941 (+ 999us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:00.643950 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:00.643953 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:00.644124 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:00.644618 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:00.644623 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:00.644626 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:00.645060 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:50:00.645356 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0619 18:50:00.646516 (+ 1160us) server_negotiation.cc:299] Negotiation successful
0619 18:50:00.646740 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 18:50:00.647807 17116 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:50:00.652356 17451 ts_tablet_manager.cc:1918] T 83a92a069f7d400c91db23e4444469fc P fb9191a13d5d4605aa972c67ee2c6f0d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:50:00.652523 17451 log.cc:1196] T 83a92a069f7d400c91db23e4444469fc P fb9191a13d5d4605aa972c67ee2c6f0d: Deleting WAL directory at /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ts-1/wal/wals/83a92a069f7d400c91db23e4444469fc
I20260619 18:50:00.653033 17451 ts_tablet_manager.cc:1939] T 83a92a069f7d400c91db23e4444469fc P fb9191a13d5d4605aa972c67ee2c6f0d: Deleting consensus metadata
I20260619 18:50:00.655074 16910 catalog_manager.cc:4841] TS fb9191a13d5d4605aa972c67ee2c6f0d (127.3.243.66:35753): tablet 83a92a069f7d400c91db23e4444469fc (table test-table [id=c9e7c345891b4cbabcd62c0ea6217d8a]) successfully deleted
I20260619 18:50:00.655573 17450 ts_tablet_manager.cc:1918] T 83a92a069f7d400c91db23e4444469fc P 0f9b804b0e4d4d98978b08e172a9d104: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:50:00.655823 17450 log.cc:1196] T 83a92a069f7d400c91db23e4444469fc P 0f9b804b0e4d4d98978b08e172a9d104: Deleting WAL directory at /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894810715282-4045-0/minicluster-data/ts-2/wal/wals/83a92a069f7d400c91db23e4444469fc
I20260619 18:50:00.656661 17450 ts_tablet_manager.cc:1939] T 83a92a069f7d400c91db23e4444469fc P 0f9b804b0e4d4d98978b08e172a9d104: Deleting consensus metadata
I20260619 18:50:00.658656 16909 catalog_manager.cc:4841] TS 0f9b804b0e4d4d98978b08e172a9d104 (127.3.243.67:33993): tablet 83a92a069f7d400c91db23e4444469fc (table test-table [id=c9e7c345891b4cbabcd62c0ea6217d8a]) successfully deleted
I20260619 18:50:00.743299 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 16961
I20260619 18:50:00.762928 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 17123
W20260619 18:50:00.762933 16911 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.243.65:39543 (error 108)
I20260619 18:50:00.765147 17423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:00.764767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.65:39543 (local address 127.0.0.1:50922)
0619 18:50:00.764915 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:00.765017 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.65:39543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260619 18:50:00.765429 16911 catalog_manager.cc:4584] TS 903297c2967c48f0a501243b94829f0d (127.3.243.65:39543): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 83a92a069f7d400c91db23e4444469fc: Network error: Client connection negotiation failed: client connection to 127.3.243.65:39543: connect: Connection refused (error 111)
I20260619 18:50:00.777487 17424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:00.763817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32850 (local address 127.3.243.66:35753)
0619 18:50:00.763995 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 18:50:00.764001 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:50:00.764022 (+ 21us) server_negotiation.cc:407] Connection header received
0619 18:50:00.764189 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:50:00.764195 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:50:00.764280 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:50:00.764388 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:50:00.766223 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:00.766944 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:00.768801 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:00.769196 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:00.769288 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:00.772268 (+ 2980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:50:00.772282 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:50:00.772296 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:50:00.772325 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:50:00.774992 (+ 2667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:00.775713 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:00.775720 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:00.775722 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:00.775840 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:00.776208 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:00.776211 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:00.776213 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:00.776543 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:50:00.776748 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0619 18:50:00.777056 (+ 308us) server_negotiation.cc:299] Negotiation successful
0619 18:50:00.777230 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 18:50:00.778156 17260 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:50:00.819067 17423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:00.818487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.65:39543 (local address 127.0.0.1:50932)
0619 18:50:00.818735 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:00.818869 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.65:39543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260619 18:50:00.820262 17423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:00.819822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.65:39543 (local address 127.0.0.1:50940)
0619 18:50:00.819979 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:00.820075 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.65:39543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260619 18:50:00.865939 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 17123
I20260619 18:50:00.880869 17423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:00.880420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.65:39543 (local address 127.0.0.1:50948)
0619 18:50:00.880623 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:00.880720 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.65:39543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260619 18:50:00.881875 17423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:00.881548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.65:39543 (local address 127.0.0.1:50958)
0619 18:50:00.881690 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:00.881763 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.65:39543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260619 18:50:00.886010 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 17270
I20260619 18:50:00.900889 17425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:00.886729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46848 (local address 127.3.243.67:33993)
0619 18:50:00.886904 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 18:50:00.886909 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:50:00.886928 (+ 19us) server_negotiation.cc:407] Connection header received
0619 18:50:00.887064 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:50:00.887070 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:50:00.887133 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:50:00.887225 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:50:00.888503 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:00.889312 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:00.890609 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:00.891047 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:00.891161 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:00.893665 (+ 2504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:50:00.893679 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:50:00.893694 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:50:00.893729 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:50:00.898132 (+ 4403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:00.898758 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:00.898765 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:00.898768 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:00.898926 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:00.899282 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:00.899286 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:00.899288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:00.899675 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:50:00.899962 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0619 18:50:00.900373 (+ 411us) server_negotiation.cc:299] Negotiation successful
0619 18:50:00.900584 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 18:50:00.901582 17395 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:50:00.962286 17423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:00.961748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.65:39543 (local address 127.0.0.1:50962)
0619 18:50:00.961964 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:00.962082 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.65:39543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260619 18:50:00.963342 17423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:00.962922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.65:39543 (local address 127.0.0.1:50972)
0619 18:50:00.963115 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:00.963208 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.65:39543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260619 18:50:01.021154 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 17270
I20260619 18:50:01.035943 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 16891
I20260619 18:50:01.037106 16954 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:50:01.148056 17423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:01.147396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.65:39543 (local address 127.0.0.1:50984)
0619 18:50:01.147720 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:01.147837 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.65:39543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199}
I20260619 18:50:01.148042 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 16891
I20260619 18:50:01.168558 4045 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 18:50:01.502549 4045 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 18:50:01.502736 4045 mini_ranger.cc:66] Stopping Ranger...
I20260619 18:50:01.836313 4045 mini_ranger.cc:68] Stopped Ranger
2026-06-19 18:50:01.836 UTC [15943] LOG: received smart shutdown request
2026-06-19 18:50:01.839 UTC [15943] LOG: background worker "logical replication launcher" (PID 15952) exited with exit code 1
2026-06-19 18:50:01.839 UTC [15947] LOG: shutting down
2026-06-19 18:50:01.978 UTC [15943] LOG: database system is shut down
2026-06-19T18:50:01Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (50124 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894810715282-4045-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:50:02 dist-test-slave-m9wg krb5kdc[17466](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:50:02 dist-test-slave-m9wg krb5kdc[17466](info): set up 2 sockets
Jun 19 18:50:02 dist-test-slave-m9wg krb5kdc[17466](info): commencing operation
krb5kdc: starting...
W20260619 18:50:03.108379 4045 mini_kdc.cc:121] Time spent starting KDC: real 0.977s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894810715282-4045-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894810715282-4045-0/krb5kdc/test-user.keytab.
Jun 19 18:50:03 dist-test-slave-m9wg krb5kdc[17466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895003, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:50:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:50:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.243.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab.
Entry for principal kudu/127.3.243.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab.
I20260619 18:50:03.316598 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894810715282-4045-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894810715282-4045-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894810715282-4045-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.126:42939
--webserver_interface=127.3.243.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab
--principal=kudu/127.3.243.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:34921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.126:42939,127.3.243.125:32927,127.3.243.124:44199
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:50:03.626516 17479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:50:03.626820 17479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:50:03.626953 17479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:50:03.634119 17479 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:50:03.634236 17479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:50:03.634315 17479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:50:03.634392 17479 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:50:03.634452 17479 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:50:03.643896 17479 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:34921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894810715282-4045-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894810715282-4045-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.243.126:42939,127.3.243.125:32927,127.3.243.124:44199
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab
--principal=kudu/127.3.243.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.126:42939
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.243.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894810715282-4045-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:50:03.645151 17479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:50:03.646428 17479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:50:03.657668 17484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:50:03.657691 17485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:50:03.657624 17487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:50:03.658591 17479 server_base.cc:1034] running on GCE node
I20260619 18:50:03.659376 17479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:50:03.660807 17479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:50:03.662005 17479 hybrid_clock.cc:648] HybridClock initialized: now 1781895003661963 us; error 35 us; skew 500 ppm
I20260619 18:50:03.662345 17479 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:50:03 dist-test-slave-m9wg krb5kdc[17466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895003, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:50:03.665494 17479 init.cc:377] Logged in from keytab as kudu/127.3.243.126@KRBTEST.COM (short username kudu)
I20260619 18:50:03.667008 17479 webserver.cc:460] Webserver started at http://127.3.243.126:43229/ using document root <none> and password file <none>
I20260619 18:50:03.667717 17479 fs_manager.cc:359] Metadata directory not provided
I20260619 18:50:03.667836 17479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:50:03.668200 17479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:50:03.671180 17479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894810715282-4045-0/minicluster-data/master-0/data/instance:
uuid: "b3fb469ed29c41ada7bf3d7d51cd308d"
format_stamp: "Formatted at 2026-06-19 18:50:03 on dist-test-slave-m9wg"
server_key: "85b8d0e037b4bb1bbbc4f20756cbbc9b"
server_key_iv: "d694b3499a0d7fa8ca05d948c11bfba3"
server_key_version: "encryptionkey@0"
I20260619 18:50:03.671958 17479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894810715282-4045-0/minicluster-data/master-0/wal/instance:
uuid: "b3fb469ed29c41ada7bf3d7d51cd308d"
format_stamp: "Formatted at 2026-06-19 18:50:03 on dist-test-slave-m9wg"
server_key: "85b8d0e037b4bb1bbbc4f20756cbbc9b"
server_key_iv: "d694b3499a0d7fa8ca05d948c11bfba3"
server_key_version: "encryptionkey@0"
I20260619 18:50:03.677090 17479 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 18:50:03.680467 17494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:50:03.681787 17479 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 18:50:03.681947 17479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894810715282-4045-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894810715282-4045-0/minicluster-data/master-0/wal
uuid: "b3fb469ed29c41ada7bf3d7d51cd308d"
format_stamp: "Formatted at 2026-06-19 18:50:03 on dist-test-slave-m9wg"
server_key: "85b8d0e037b4bb1bbbc4f20756cbbc9b"
server_key_iv: "d694b3499a0d7fa8ca05d948c11bfba3"
server_key_version: "encryptionkey@0"
I20260619 18:50:03.682133 17479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894810715282-4045-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894810715282-4045-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894810715282-4045-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:50:03.699169 17479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:50:03.703302 17479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:50:03.703547 17479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:50:03.721288 17479 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.126:42939
I20260619 18:50:03.721320 17545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.126:42939 every 8 connection(s)
I20260619 18:50:03.722563 17479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
I20260619 18:50:03.725807 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 17479
I20260619 18:50:03.725996 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894810715282-4045-0/minicluster-data/master-0/wal/instance
I20260619 18:50:03.726464 17546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:50:03.726649 4045 external_mini_cluster.cc:1428] Setting key af92faca1d9e913191eed82d7ce196b1
I20260619 18:50:03.733151 17546 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.243.126" port: 42939 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.3.243.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.125@KRBTEST.COM" created.
Jun 19 18:50:03 dist-test-slave-m9wg krb5kdc[17466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895003, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.126@KRBTEST.COM for kudu/127.3.243.126@KRBTEST.COM
I20260619 18:50:03.757827 17549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:03.736716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:42939 (local address 127.0.0.1:47226)
0619 18:50:03.737639 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:03.737693 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 18:50:03.740466 (+ 2773us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:50:03.741400 (+ 934us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:50:03.741405 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:50:03.741983 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:50:03.742325 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:50:03.742343 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:50:03.744529 (+ 2186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:50:03.744533 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:50:03.745613 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:50:03.745626 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:50:03.746842 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:50:03.746846 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:50:03.746963 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:03.747606 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:50:03.747628 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:50:03.750851 (+ 3223us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:50:03.753803 (+ 2952us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:50:03.753815 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:50:03.753828 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:50:03.754320 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:50:03.754697 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:50:03.754701 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:50:03.754704 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:50:03.754895 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:50:03.755636 (+ 741us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:50:03.755644 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:50:03.755886 (+ 242us) client_negotiation.cc:769] Sending connection context
0619 18:50:03.756177 (+ 291us) client_negotiation.cc:240] Negotiation successful
0619 18:50:03.756815 (+ 638us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"mutex_wait_us":199,"thread_start_us":355,"threads_started":1}
I20260619 18:50:03.757861 17550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:03.736715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47226 (local address 127.3.243.126:42939)
0619 18:50:03.740649 (+ 3934us) server_negotiation.cc:206] Beginning negotiation
0619 18:50:03.740657 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:50:03.740723 (+ 66us) server_negotiation.cc:407] Connection header received
0619 18:50:03.740886 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:50:03.740892 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:50:03.741182 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:50:03.741388 (+ 206us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:50:03.742515 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:03.744363 (+ 1848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:03.745757 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:03.746637 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:03.746837 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:03.751079 (+ 4242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:50:03.751087 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:50:03.751099 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:50:03.751128 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:50:03.753682 (+ 2554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:03.754427 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:03.754433 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:03.754436 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:03.754578 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:03.755008 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:03.755011 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:03.755013 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:03.755507 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:50:03.755659 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 18:50:03.756274 (+ 615us) server_negotiation.cc:299] Negotiation successful
0619 18:50:03.756702 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":3736,"thread_start_us":3531,"threads_started":1}
I20260619 18:50:03.760771 17546 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.243.125" port: 32927 } has no permanent_uuid. Determining permanent_uuid...
I20260619 18:50:03.761865 17549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:03.761424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.125:32927 (local address 127.0.0.1:56608)
0619 18:50:03.761597 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:03.761731 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.125:32927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260619 18:50:03.762143 17497 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.243.125:32927: connect: Connection refused (error 111)
I20260619 18:50:03.763676 17549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:03.763287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.125:32927 (local address 127.0.0.1:56622)
0619 18:50:03.763449 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:03.763529 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.125:32927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260619 18:50:03.763870 17546 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.243.125:32927: Network error: Client connection negotiation failed: client connection to 127.3.243.125:32927: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.125.keytab.
Entry for principal kudu/127.3.243.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.125.keytab.
I20260619 18:50:03.776340 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894810715282-4045-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894810715282-4045-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894810715282-4045-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894810715282-4045-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.125:32927
--webserver_interface=127.3.243.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.125.keytab
--principal=kudu/127.3.243.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:34921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.126:42939,127.3.243.125:32927,127.3.243.124:44199
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 18:50:03.813225 17546 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.243.125" port: 32927 } attempt: 1
I20260619 18:50:03.814500 17549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:03.813951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.125:32927 (local address 127.0.0.1:56624)
0619 18:50:03.814178 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:03.814330 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.125:32927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260619 18:50:03.815485 17549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:03.815090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.125:32927 (local address 127.0.0.1:56632)
0619 18:50:03.815242 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:03.815326 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.125:32927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260619 18:50:03.815673 17546 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.243.125:32927: Network error: Client connection negotiation failed: client connection to 127.3.243.125:32927: connect: Connection refused (error 111)
I20260619 18:50:03.883945 17546 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.243.125" port: 32927 } attempt: 2
I20260619 18:50:03.885221 17549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:03.884660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.125:32927 (local address 127.0.0.1:56640)
0619 18:50:03.884914 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:03.885052 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.125:32927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260619 18:50:03.886279 17549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:03.885887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.125:32927 (local address 127.0.0.1:56652)
0619 18:50:03.886029 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:03.886125 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.125:32927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260619 18:50:03.886493 17546 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.243.125:32927: Network error: Client connection negotiation failed: client connection to 127.3.243.125:32927: connect: Connection refused (error 111)
I20260619 18:50:03.977764 17546 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.243.125" port: 32927 } attempt: 3
I20260619 18:50:03.978967 17549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:03.978454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.125:32927 (local address 127.0.0.1:56656)
0619 18:50:03.978681 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:03.978809 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.125:32927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260619 18:50:03.980008 17549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:03.979583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.125:32927 (local address 127.0.0.1:56668)
0619 18:50:03.979736 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:03.979828 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.125:32927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 18:50:03.980262 17546 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.243.125:32927: Network error: Client connection negotiation failed: client connection to 127.3.243.125:32927: connect: Connection refused (error 111)
W20260619 18:50:04.081605 17553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:50:04.081868 17553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:50:04.081997 17553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:50:04.089062 17553 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:50:04.089166 17553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:50:04.089238 17553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:50:04.089308 17553 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:50:04.089368 17553 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:50:04.098817 17553 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:34921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894810715282-4045-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894810715282-4045-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.243.126:42939,127.3.243.125:32927,127.3.243.124:44199
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.125.keytab
--principal=kudu/127.3.243.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.125:32927
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894810715282-4045-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.243.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894810715282-4045-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:50:04.100198 17553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:50:04.101423 17553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:50:04.112087 17559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:50:04.112236 17558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:50:04.112347 17561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:50:04.112768 17553 server_base.cc:1034] running on GCE node
I20260619 18:50:04.113454 17553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:50:04.114735 17553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:50:04.115921 17553 hybrid_clock.cc:648] HybridClock initialized: now 1781895004115864 us; error 51 us; skew 500 ppm
I20260619 18:50:04.116258 17553 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:50:04 dist-test-slave-m9wg krb5kdc[17466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895004, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:50:04.119057 17553 init.cc:377] Logged in from keytab as kudu/127.3.243.125@KRBTEST.COM (short username kudu)
I20260619 18:50:04.120445 17553 webserver.cc:460] Webserver started at http://127.3.243.125:43573/ using document root <none> and password file <none>
I20260619 18:50:04.121073 17553 fs_manager.cc:359] Metadata directory not provided
I20260619 18:50:04.121183 17553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:50:04.121443 17553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:50:04.123497 17546 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.243.125" port: 32927 } attempt: 4
I20260619 18:50:04.124505 17549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:04.123991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.125:32927 (local address 127.0.0.1:56678)
0619 18:50:04.124289 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:04.124368 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.125:32927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134}
I20260619 18:50:04.124281 17553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894810715282-4045-0/minicluster-data/master-1/data/instance:
uuid: "f33d9a082430441494dbb1e660e29173"
format_stamp: "Formatted at 2026-06-19 18:50:04 on dist-test-slave-m9wg"
server_key: "260903cb718a69af33e6b463234b5626"
server_key_iv: "b713784fa91c2527488efa0ed8263cb4"
server_key_version: "encryptionkey@0"
I20260619 18:50:04.125013 17553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894810715282-4045-0/minicluster-data/master-1/wal/instance:
uuid: "f33d9a082430441494dbb1e660e29173"
format_stamp: "Formatted at 2026-06-19 18:50:04 on dist-test-slave-m9wg"
server_key: "260903cb718a69af33e6b463234b5626"
server_key_iv: "b713784fa91c2527488efa0ed8263cb4"
server_key_version: "encryptionkey@0"
I20260619 18:50:04.125385 17549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:04.125056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.125:32927 (local address 127.0.0.1:56686)
0619 18:50:04.125189 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:04.125272 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.125:32927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260619 18:50:04.125545 17546 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.243.125:32927: Network error: Client connection negotiation failed: client connection to 127.3.243.125:32927: connect: Connection refused (error 111)
I20260619 18:50:04.129606 17553 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 18:50:04.132797 17568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:50:04.133955 17553 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:50:04.134090 17553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894810715282-4045-0/minicluster-data/master-1/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894810715282-4045-0/minicluster-data/master-1/wal
uuid: "f33d9a082430441494dbb1e660e29173"
format_stamp: "Formatted at 2026-06-19 18:50:04 on dist-test-slave-m9wg"
server_key: "260903cb718a69af33e6b463234b5626"
server_key_iv: "b713784fa91c2527488efa0ed8263cb4"
server_key_version: "encryptionkey@0"
I20260619 18:50:04.134243 17553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894810715282-4045-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894810715282-4045-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894810715282-4045-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:50:04.154675 17553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:50:04.158411 17553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:50:04.158656 17553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:50:04.174129 17553 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.125:32927
I20260619 18:50:04.174152 17619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.125:32927 every 8 connection(s)
I20260619 18:50:04.175424 17553 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894810715282-4045-0/minicluster-data/master-1/data/info.pb
I20260619 18:50:04.178917 17620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:50:04.185389 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 17553
I20260619 18:50:04.185580 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894810715282-4045-0/minicluster-data/master-1/wal/instance
I20260619 18:50:04.185173 17620 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.243.126" port: 42939 } has no permanent_uuid. Determining permanent_uuid...
I20260619 18:50:04.186163 4045 external_mini_cluster.cc:1428] Setting key 0c2329e15ba0438519cc9e4909617c0c
Jun 19 18:50:04 dist-test-slave-m9wg krb5kdc[17466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895004, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.125@KRBTEST.COM for kudu/127.3.243.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.243.124@KRBTEST.COM; defaulting to no policy
I20260619 18:50:04.205418 17550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:04.188194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47242 (local address 127.3.243.126:42939)
0619 18:50:04.188385 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 18:50:04.188390 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:50:04.189499 (+ 1109us) server_negotiation.cc:407] Connection header received
0619 18:50:04.191837 (+ 2338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:50:04.191841 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:50:04.191907 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:50:04.191992 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:50:04.193839 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:04.194508 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:04.196304 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:04.196742 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:04.196858 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:04.200040 (+ 3182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:50:04.200045 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:50:04.200048 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:50:04.200081 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:50:04.202582 (+ 2501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:04.203261 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:04.203265 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:04.203268 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:04.203384 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:04.203769 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:04.203772 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:04.203774 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:04.204073 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:50:04.204228 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0619 18:50:04.204886 (+ 658us) server_negotiation.cc:299] Negotiation successful
0619 18:50:04.205119 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 18:50:04.206420 17622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:04.188616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:42939 (local address 127.0.0.1:47242)
0619 18:50:04.189333 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:04.189379 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 18:50:04.191572 (+ 2193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:50:04.192251 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:50:04.192262 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:50:04.192938 (+ 676us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:50:04.193658 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:50:04.193680 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:50:04.194662 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:50:04.194667 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:50:04.196140 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:50:04.196179 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:50:04.196958 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:50:04.196966 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:50:04.197182 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:04.197773 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:50:04.197794 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:50:04.199809 (+ 2015us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:50:04.202714 (+ 2905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:50:04.202723 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:50:04.202739 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:50:04.203142 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:50:04.203488 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:50:04.203493 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:50:04.203496 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:50:04.203660 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:50:04.204213 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:50:04.204220 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:50:04.204535 (+ 315us) client_negotiation.cc:769] Sending connection context
0619 18:50:04.204760 (+ 225us) client_negotiation.cc:240] Negotiation successful
0619 18:50:04.205214 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":275,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.124@KRBTEST.COM" created.
I20260619 18:50:04.208092 17620 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.243.125" port: 32927 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 18:50:04 dist-test-slave-m9wg krb5kdc[17466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895004, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.125@KRBTEST.COM for kudu/127.3.243.125@KRBTEST.COM
I20260619 18:50:04.221446 17622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:04.208882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.125:32927 (local address 127.0.0.1:56700)
0619 18:50:04.209041 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:04.209058 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 18:50:04.209153 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:50:04.210070 (+ 917us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:50:04.210075 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:50:04.210440 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:50:04.210554 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:50:04.210561 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:50:04.211853 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:50:04.211857 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:50:04.212680 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:50:04.212691 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:50:04.213405 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:50:04.213411 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:50:04.213580 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:04.214043 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:50:04.214059 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:50:04.215987 (+ 1928us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:50:04.219376 (+ 3389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:50:04.219379 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:50:04.219382 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:50:04.219719 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:50:04.220075 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:50:04.220078 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:50:04.220080 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:50:04.220256 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:50:04.220837 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:50:04.220840 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:50:04.220940 (+ 100us) client_negotiation.cc:769] Sending connection context
0619 18:50:04.221041 (+ 101us) client_negotiation.cc:240] Negotiation successful
0619 18:50:04.221210 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260619 18:50:04.221791 17624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:04.209120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56700 (local address 127.3.243.125:32927)
0619 18:50:04.209705 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0619 18:50:04.209710 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:50:04.209733 (+ 23us) server_negotiation.cc:407] Connection header received
0619 18:50:04.209771 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:50:04.209777 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:50:04.209937 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:50:04.210059 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:50:04.210696 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:04.211735 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:04.212795 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:04.213285 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:04.213386 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:04.216204 (+ 2818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:50:04.216216 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:50:04.216231 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:50:04.216268 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:50:04.219257 (+ 2989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:04.219837 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:04.219846 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:04.219849 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:04.219987 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:04.220345 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:04.220350 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:04.220353 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:04.220731 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:50:04.220929 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0619 18:50:04.221182 (+ 253us) server_negotiation.cc:299] Negotiation successful
0619 18:50:04.221408 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":329,"threads_started":1}
I20260619 18:50:04.224031 17620 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.243.124" port: 44199 } has no permanent_uuid. Determining permanent_uuid...
I20260619 18:50:04.225078 17622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:04.224594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.124:44199 (local address 127.0.0.1:51332)
0619 18:50:04.224774 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:04.224906 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.124:44199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260619 18:50:04.225310 17572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.243.124:44199: connect: Connection refused (error 111)
I20260619 18:50:04.226763 17622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:04.226390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.124:44199 (local address 127.0.0.1:51334)
0619 18:50:04.226544 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:04.226636 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.124:44199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260619 18:50:04.226981 17620 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.243.124:44199: Network error: Client connection negotiation failed: client connection to 127.3.243.124:44199: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.124.keytab.
Entry for principal kudu/127.3.243.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.124.keytab.
I20260619 18:50:04.235229 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894810715282-4045-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894810715282-4045-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894810715282-4045-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894810715282-4045-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.124:44199
--webserver_interface=127.3.243.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.124.keytab
--principal=kudu/127.3.243.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:34921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.126:42939,127.3.243.125:32927,127.3.243.124:44199
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 18:50:04.276283 17620 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.243.124" port: 44199 } attempt: 1
I20260619 18:50:04.277496 17622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:04.276951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.124:44199 (local address 127.0.0.1:51336)
0619 18:50:04.277186 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:04.277316 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.124:44199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260619 18:50:04.278590 17622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:04.278165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.124:44199 (local address 127.0.0.1:51350)
0619 18:50:04.278344 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:04.278442 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.124:44199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260619 18:50:04.278795 17620 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.243.124:44199: Network error: Client connection negotiation failed: client connection to 127.3.243.124:44199: connect: Connection refused (error 111)
I20260619 18:50:04.347000 17620 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.243.124" port: 44199 } attempt: 2
I20260619 18:50:04.348217 17622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:04.347620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.124:44199 (local address 127.0.0.1:51358)
0619 18:50:04.347856 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:04.347978 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.124:44199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260619 18:50:04.349220 17622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:04.348832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.124:44199 (local address 127.0.0.1:51372)
0619 18:50:04.348964 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:04.349047 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.124:44199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260619 18:50:04.349413 17620 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.243.124:44199: Network error: Client connection negotiation failed: client connection to 127.3.243.124:44199: connect: Connection refused (error 111)
I20260619 18:50:04.424793 17546 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.243.125" port: 32927 } attempt: 5
Jun 19 18:50:04 dist-test-slave-m9wg krb5kdc[17466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895003, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.126@KRBTEST.COM for kudu/127.3.243.125@KRBTEST.COM
I20260619 18:50:04.438956 17624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:04.425643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56714 (local address 127.3.243.125:32927)
0619 18:50:04.425857 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0619 18:50:04.425861 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:50:04.425885 (+ 24us) server_negotiation.cc:407] Connection header received
0619 18:50:04.426076 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:50:04.426081 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:50:04.426166 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:50:04.426253 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:50:04.427309 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:04.427997 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:04.429830 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:04.430284 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:04.430381 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:04.433876 (+ 3495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:50:04.433882 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:50:04.433885 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:50:04.433919 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:50:04.436415 (+ 2496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:04.437067 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:04.437071 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:04.437073 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:04.437190 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:04.437604 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:04.437609 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:04.437612 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:04.437909 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:50:04.438043 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 18:50:04.438397 (+ 354us) server_negotiation.cc:299] Negotiation successful
0619 18:50:04.438640 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260619 18:50:04.438957 17549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:04.425483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.125:32927 (local address 127.0.0.1:56714)
0619 18:50:04.425769 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:04.425793 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 18:50:04.425938 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:50:04.426320 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:50:04.426325 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:50:04.426862 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:50:04.427137 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:50:04.427153 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:50:04.428180 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:50:04.428186 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:50:04.429655 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:50:04.429677 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:50:04.430457 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:50:04.430463 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:50:04.430661 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:04.431292 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:50:04.431315 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:50:04.433688 (+ 2373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:50:04.436553 (+ 2865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:50:04.436558 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:50:04.436563 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:50:04.436959 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:50:04.437317 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:50:04.437322 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:50:04.437326 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:50:04.437495 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:50:04.438017 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:50:04.438022 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:50:04.438173 (+ 151us) client_negotiation.cc:769] Sending connection context
0619 18:50:04.438287 (+ 114us) client_negotiation.cc:240] Negotiation successful
0619 18:50:04.438606 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93}
I20260619 18:50:04.440045 17546 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.243.124" port: 44199 } has no permanent_uuid. Determining permanent_uuid...
I20260619 18:50:04.440619 17620 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.243.124" port: 44199 } attempt: 3
I20260619 18:50:04.441383 17549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:04.440979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.124:44199 (local address 127.0.0.1:51388)
0619 18:50:04.441150 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:04.441236 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.124:44199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260619 18:50:04.441603 17622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:04.441120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.124:44199 (local address 127.0.0.1:51402)
0619 18:50:04.441349 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:04.441434 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.124:44199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
I20260619 18:50:04.442363 17549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:04.441955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.124:44199 (local address 127.0.0.1:51406)
0619 18:50:04.442140 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:04.442243 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.124:44199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260619 18:50:04.442608 17546 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.243.124:44199: Network error: Client connection negotiation failed: client connection to 127.3.243.124:44199: connect: Connection refused (error 111)
I20260619 18:50:04.442811 17622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:04.442245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.124:44199 (local address 127.0.0.1:51418)
0619 18:50:04.442464 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:04.442671 (+ 207us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.124:44199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
W20260619 18:50:04.443027 17620 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.243.124:44199: Network error: Client connection negotiation failed: client connection to 127.3.243.124:44199: connect: Connection refused (error 111)
I20260619 18:50:04.493882 17546 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.243.124" port: 44199 } attempt: 1
I20260619 18:50:04.495042 17549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:04.494560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.124:44199 (local address 127.0.0.1:51434)
0619 18:50:04.494762 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:04.494865 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.124:44199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260619 18:50:04.495963 17549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:04.495649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.124:44199 (local address 127.0.0.1:51450)
0619 18:50:04.495787 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:04.495858 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.124:44199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260619 18:50:04.496181 17546 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.243.124:44199: Network error: Client connection negotiation failed: client connection to 127.3.243.124:44199: connect: Connection refused (error 111)
W20260619 18:50:04.548305 17627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:50:04.548611 17627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:50:04.548746 17627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:50:04.556033 17627 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:50:04.556170 17627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:50:04.556239 17627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:50:04.556313 17627 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:50:04.556371 17627 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:50:04.564379 17546 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.243.124" port: 44199 } attempt: 2
I20260619 18:50:04.565506 17549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:04.565011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.124:44199 (local address 127.0.0.1:51462)
0619 18:50:04.565235 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:04.565333 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.124:44199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260619 18:50:04.566447 17549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:04.566121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.124:44199 (local address 127.0.0.1:51474)
0619 18:50:04.566257 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:04.566326 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.124:44199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260619 18:50:04.566434 17627 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:34921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894810715282-4045-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894810715282-4045-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.243.126:42939,127.3.243.125:32927,127.3.243.124:44199
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.124.keytab
--principal=kudu/127.3.243.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.124:44199
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894810715282-4045-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.243.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894810715282-4045-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
W20260619 18:50:04.566641 17546 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.243.124:44199: Network error: Client connection negotiation failed: client connection to 127.3.243.124:44199: connect: Connection refused (error 111)
I20260619 18:50:04.567827 17627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:50:04.569186 17627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:50:04.580909 17633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:50:04.581157 17635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:50:04.580884 17632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:50:04.581655 17627 server_base.cc:1034] running on GCE node
I20260619 18:50:04.582458 17627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:50:04.583969 17627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:50:04.585180 17627 hybrid_clock.cc:648] HybridClock initialized: now 1781895004585123 us; error 47 us; skew 500 ppm
I20260619 18:50:04.585553 17627 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 18:50:04.586227 17620 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.243.124" port: 44199 } attempt: 4
I20260619 18:50:04.587277 17622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:04.586803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.124:44199 (local address 127.0.0.1:51488)
0619 18:50:04.587041 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:04.587123 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.124:44199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260619 18:50:04.588306 17622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:04.587920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.124:44199 (local address 127.0.0.1:51492)
0619 18:50:04.588072 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:04.588179 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.124:44199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
Jun 19 18:50:04 dist-test-slave-m9wg krb5kdc[17466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895004, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260619 18:50:04.588562 17620 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.243.124:44199: Network error: Client connection negotiation failed: client connection to 127.3.243.124:44199: connect: Connection refused (error 111)
I20260619 18:50:04.588912 17627 init.cc:377] Logged in from keytab as kudu/127.3.243.124@KRBTEST.COM (short username kudu)
I20260619 18:50:04.590389 17627 webserver.cc:460] Webserver started at http://127.3.243.124:33135/ using document root <none> and password file <none>
I20260619 18:50:04.591066 17627 fs_manager.cc:359] Metadata directory not provided
I20260619 18:50:04.591173 17627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:50:04.591462 17627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:50:04.594545 17627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894810715282-4045-0/minicluster-data/master-2/data/instance:
uuid: "533331520ff242dfb22983d8aad9617c"
format_stamp: "Formatted at 2026-06-19 18:50:04 on dist-test-slave-m9wg"
server_key: "f31974ad3b5feeed446c81a0b447f41b"
server_key_iv: "8e7f816e0a7c68d9a38674f34d34fed5"
server_key_version: "encryptionkey@0"
I20260619 18:50:04.595346 17627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894810715282-4045-0/minicluster-data/master-2/wal/instance:
uuid: "533331520ff242dfb22983d8aad9617c"
format_stamp: "Formatted at 2026-06-19 18:50:04 on dist-test-slave-m9wg"
server_key: "f31974ad3b5feeed446c81a0b447f41b"
server_key_iv: "8e7f816e0a7c68d9a38674f34d34fed5"
server_key_version: "encryptionkey@0"
I20260619 18:50:04.600531 17627 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260619 18:50:04.603844 17642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:50:04.605325 17627 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:50:04.605494 17627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894810715282-4045-0/minicluster-data/master-2/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894810715282-4045-0/minicluster-data/master-2/wal
uuid: "533331520ff242dfb22983d8aad9617c"
format_stamp: "Formatted at 2026-06-19 18:50:04 on dist-test-slave-m9wg"
server_key: "f31974ad3b5feeed446c81a0b447f41b"
server_key_iv: "8e7f816e0a7c68d9a38674f34d34fed5"
server_key_version: "encryptionkey@0"
I20260619 18:50:04.605675 17627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894810715282-4045-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894810715282-4045-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894810715282-4045-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:50:04.626231 17627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:50:04.630494 17627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:50:04.630760 17627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:50:04.651441 17627 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.124:44199
I20260619 18:50:04.651479 17693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.124:44199 every 8 connection(s)
I20260619 18:50:04.652891 17627 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894810715282-4045-0/minicluster-data/master-2/data/info.pb
I20260619 18:50:04.654793 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 17627
I20260619 18:50:04.655018 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894810715282-4045-0/minicluster-data/master-2/wal/instance
I20260619 18:50:04.655625 4045 external_mini_cluster.cc:1428] Setting key d9335e871175c4c76e46ab8a9e6dde31
I20260619 18:50:04.657595 17694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:50:04.664811 17694 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.243.126" port: 42939 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 18:50:04 dist-test-slave-m9wg krb5kdc[17466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895003, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.124@KRBTEST.COM
I20260619 18:50:04.672839 17546 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.243.124" port: 44199 } attempt: 3
Jun 19 18:50:04 dist-test-slave-m9wg krb5kdc[17466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895004, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.124@KRBTEST.COM for kudu/127.3.243.126@KRBTEST.COM
I20260619 18:50:04.679517 17697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:04.657520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51504 (local address 127.3.243.124:44199)
0619 18:50:04.658308 (+ 788us) server_negotiation.cc:206] Beginning negotiation
0619 18:50:04.658320 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 18:50:04.658359 (+ 39us) server_negotiation.cc:407] Connection header received
0619 18:50:04.659930 (+ 1571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:50:04.659952 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:50:04.660401 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:50:04.660777 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:50:04.662217 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:04.663509 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:04.664824 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:04.665518 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:04.665680 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:04.670315 (+ 4635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:50:04.670322 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:50:04.670332 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:50:04.670369 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:50:04.673747 (+ 3378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:04.674661 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:04.674670 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:04.674680 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:04.674938 (+ 258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:04.675390 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:04.675399 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:04.675401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:04.676178 (+ 777us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:50:04.676480 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0619 18:50:04.677746 (+ 1266us) server_negotiation.cc:299] Negotiation successful
0619 18:50:04.678133 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":346,"threads_started":1}
Jun 19 18:50:04 dist-test-slave-m9wg krb5kdc[17466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895003, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.126@KRBTEST.COM for kudu/127.3.243.124@KRBTEST.COM
I20260619 18:50:04.683771 17698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:04.667449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:42939 (local address 127.0.0.1:47254)
0619 18:50:04.668227 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:04.668267 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 18:50:04.668416 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:50:04.668780 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:50:04.668786 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:50:04.669364 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:50:04.669617 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:50:04.669626 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:50:04.670767 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:50:04.670772 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:50:04.672351 (+ 1579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:50:04.672366 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:50:04.673252 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:50:04.673260 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:50:04.673446 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:04.674130 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:50:04.674154 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:50:04.677071 (+ 2917us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:50:04.681109 (+ 4038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:50:04.681120 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:50:04.681128 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:50:04.681523 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:50:04.681928 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:50:04.681934 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:50:04.681938 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:50:04.682068 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:50:04.682656 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:50:04.682663 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:50:04.682901 (+ 238us) client_negotiation.cc:769] Sending connection context
0619 18:50:04.683080 (+ 179us) client_negotiation.cc:240] Negotiation successful
0619 18:50:04.683407 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":393,"threads_started":1}
I20260619 18:50:04.683909 17550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:04.667393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47254 (local address 127.3.243.126:42939)
0619 18:50:04.667576 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 18:50:04.667580 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:50:04.668355 (+ 775us) server_negotiation.cc:407] Connection header received
0619 18:50:04.668549 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:50:04.668554 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:50:04.668619 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:50:04.668706 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:50:04.669767 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:04.670611 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:04.672491 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:04.673018 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:04.673179 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:04.677252 (+ 4073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:50:04.677259 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:50:04.677267 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:50:04.677306 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:50:04.680986 (+ 3680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:04.681677 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:04.681683 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:04.681686 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:04.681827 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:04.682179 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:04.682184 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:04.682187 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:04.682503 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:50:04.682657 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0619 18:50:04.683242 (+ 585us) server_negotiation.cc:299] Negotiation successful
0619 18:50:04.683494 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 18:50:04.685600 17694 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.243.125" port: 32927 } has no permanent_uuid. Determining permanent_uuid...
I20260619 18:50:04.689512 17549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:04.673394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.124:44199 (local address 127.0.0.1:51506)
0619 18:50:04.673678 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:04.673697 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 18:50:04.673796 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:50:04.674983 (+ 1187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:50:04.674989 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:50:04.675428 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:50:04.675652 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:50:04.675662 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:50:04.676921 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:50:04.676927 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:50:04.678215 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:50:04.678231 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:50:04.679123 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:50:04.679127 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:50:04.679256 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:04.679824 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:50:04.679846 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:50:04.682179 (+ 2333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:50:04.687075 (+ 4896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:50:04.687084 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:50:04.687088 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:50:04.687457 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:50:04.687890 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:50:04.687895 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:50:04.687898 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:50:04.688050 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:50:04.688723 (+ 673us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:50:04.688728 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:50:04.688884 (+ 156us) client_negotiation.cc:769] Sending connection context
0619 18:50:04.689009 (+ 125us) client_negotiation.cc:240] Negotiation successful
0619 18:50:04.689228 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158}
I20260619 18:50:04.690379 17699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:04.673687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51506 (local address 127.3.243.124:44199)
0619 18:50:04.674643 (+ 956us) server_negotiation.cc:206] Beginning negotiation
0619 18:50:04.674648 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:50:04.674669 (+ 21us) server_negotiation.cc:407] Connection header received
0619 18:50:04.674714 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:50:04.674718 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:50:04.674799 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:50:04.674928 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:50:04.675815 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:04.676793 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:04.678372 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:04.678977 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:04.679107 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:04.682412 (+ 3305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:50:04.682418 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:50:04.682422 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:50:04.682454 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:50:04.686955 (+ 4501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:04.687595 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:04.687600 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:04.687603 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:04.687758 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:04.688227 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:04.688231 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:04.688238 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:04.688571 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:50:04.689854 (+ 1283us) server_negotiation.cc:1035] Waiting for connection context
0619 18:50:04.689958 (+ 104us) server_negotiation.cc:299] Negotiation successful
0619 18:50:04.690104 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":798,"thread_start_us":382,"threads_started":1}
Jun 19 18:50:04 dist-test-slave-m9wg krb5kdc[17466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895003, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.126@KRBTEST.COM
Jun 19 18:50:04 dist-test-slave-m9wg krb5kdc[17466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895004, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.124@KRBTEST.COM for kudu/127.3.243.125@KRBTEST.COM
I20260619 18:50:04.695956 17546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3fb469ed29c41ada7bf3d7d51cd308d: Bootstrap starting.
I20260619 18:50:04.699944 17546 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b3fb469ed29c41ada7bf3d7d51cd308d: Neither blocks nor log segments found. Creating new log.
I20260619 18:50:04.700242 17698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:04.686261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.125:32927 (local address 127.0.0.1:56716)
0619 18:50:04.686405 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:04.686422 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 18:50:04.686508 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:50:04.687598 (+ 1090us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:50:04.687603 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:50:04.688077 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:50:04.688298 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:50:04.688310 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:50:04.689618 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:50:04.689623 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:50:04.690921 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:50:04.690935 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:50:04.691915 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:50:04.691921 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:50:04.692097 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:04.692638 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:50:04.692658 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:50:04.694917 (+ 2259us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:50:04.697796 (+ 2879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:50:04.697802 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:50:04.697806 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:50:04.698243 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:50:04.698659 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:50:04.698664 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:50:04.698667 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:50:04.698822 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:50:04.699369 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:50:04.699373 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:50:04.699505 (+ 132us) client_negotiation.cc:769] Sending connection context
0619 18:50:04.699622 (+ 117us) client_negotiation.cc:240] Negotiation successful
0619 18:50:04.699875 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260619 18:50:04.700675 17624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:04.686352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56716 (local address 127.3.243.125:32927)
0619 18:50:04.686530 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 18:50:04.686536 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:50:04.686554 (+ 18us) server_negotiation.cc:407] Connection header received
0619 18:50:04.686609 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:50:04.686614 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:50:04.686682 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:50:04.686776 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:50:04.688569 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:04.689447 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:04.691109 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:04.691755 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:04.691886 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:04.695095 (+ 3209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:50:04.695100 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:50:04.695103 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:50:04.695136 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:50:04.697678 (+ 2542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:04.698360 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:04.698365 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:04.698368 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:04.698519 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:04.698939 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:04.698944 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:04.698947 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:04.699263 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:50:04.699386 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 18:50:04.699723 (+ 337us) server_negotiation.cc:299] Negotiation successful
0619 18:50:04.699920 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 18:50:04.701081 17550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:04.683014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47260 (local address 127.3.243.126:42939)
0619 18:50:04.684128 (+ 1114us) server_negotiation.cc:206] Beginning negotiation
0619 18:50:04.684133 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:50:04.684169 (+ 36us) server_negotiation.cc:407] Connection header received
0619 18:50:04.684216 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:50:04.684221 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:50:04.684281 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:50:04.684375 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:50:04.686198 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:04.687367 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:04.689068 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:04.689723 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:04.689847 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:04.694668 (+ 4821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:50:04.694674 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:50:04.694678 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:50:04.694711 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:50:04.697928 (+ 3217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:04.698624 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:04.698630 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:04.698634 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:04.698765 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:04.699200 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:04.699205 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:04.699208 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:04.699546 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:50:04.699830 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0619 18:50:04.700541 (+ 711us) server_negotiation.cc:299] Negotiation successful
0619 18:50:04.700765 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1016,"thread_start_us":5602,"threads_started":1}
I20260619 18:50:04.701478 17694 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.243.124" port: 44199 } has no permanent_uuid. Determining permanent_uuid...
I20260619 18:50:04.701704 17546 log.cc:824] T 00000000000000000000000000000000 P b3fb469ed29c41ada7bf3d7d51cd308d: Log is configured to *not* fsync() on all Append() calls
I20260619 18:50:04.704941 17546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3fb469ed29c41ada7bf3d7d51cd308d: No bootstrap required, opened a new log
Jun 19 18:50:04 dist-test-slave-m9wg krb5kdc[17466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895004, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.124@KRBTEST.COM for kudu/127.3.243.124@KRBTEST.COM
Jun 19 18:50:04 dist-test-slave-m9wg krb5kdc[17466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895003, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.125@KRBTEST.COM
I20260619 18:50:04.714995 17698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:04.702214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.124:44199 (local address 127.0.0.1:51514)
0619 18:50:04.702534 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:04.702551 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 18:50:04.702641 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:50:04.702978 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:50:04.702983 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:50:04.703387 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:50:04.703563 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:50:04.703574 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:50:04.704844 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:50:04.704850 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:50:04.706117 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:50:04.706128 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:50:04.706773 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:50:04.706779 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:50:04.706974 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:04.707472 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:50:04.707494 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:50:04.709838 (+ 2344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:50:04.712557 (+ 2719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:50:04.712563 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:50:04.712567 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:50:04.712921 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:50:04.713314 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:50:04.713319 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:50:04.713323 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:50:04.713491 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:50:04.713943 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:50:04.713947 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:50:04.714083 (+ 136us) client_negotiation.cc:769] Sending connection context
0619 18:50:04.714205 (+ 122us) client_negotiation.cc:240] Negotiation successful
0619 18:50:04.714495 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"mutex_wait_us":5}
I20260619 18:50:04.714970 17699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:04.702399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51514 (local address 127.3.243.124:44199)
0619 18:50:04.702773 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0619 18:50:04.702777 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:50:04.702786 (+ 9us) server_negotiation.cc:407] Connection header received
0619 18:50:04.702813 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:50:04.702816 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:50:04.702868 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:50:04.702938 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:50:04.703704 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:04.704711 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:04.706235 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:04.706636 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:04.706761 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:04.710006 (+ 3245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:50:04.710011 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:50:04.710014 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:50:04.710045 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:50:04.712448 (+ 2403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:04.713045 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:04.713050 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:04.713053 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:04.713197 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:04.713594 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:04.713599 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:04.713602 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:04.713835 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:50:04.713984 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 18:50:04.714317 (+ 333us) server_negotiation.cc:299] Negotiation successful
0619 18:50:04.714620 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":157,"server-negotiator.queue_time_us":303}
I20260619 18:50:04.715034 17546 raft_consensus.cc:348] T 00000000000000000000000000000000 P b3fb469ed29c41ada7bf3d7d51cd308d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3fb469ed29c41ada7bf3d7d51cd308d" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 42939 } } peers { permanent_uuid: "f33d9a082430441494dbb1e660e29173" member_type: VOTER last_known_addr { host: "127.3.243.125" port: 32927 } } peers { permanent_uuid: "533331520ff242dfb22983d8aad9617c" member_type: VOTER last_known_addr { host: "127.3.243.124" port: 44199 } }
I20260619 18:50:04.715405 17546 raft_consensus.cc:374] T 00000000000000000000000000000000 P b3fb469ed29c41ada7bf3d7d51cd308d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:50:04.715559 17546 raft_consensus.cc:729] T 00000000000000000000000000000000 P b3fb469ed29c41ada7bf3d7d51cd308d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3fb469ed29c41ada7bf3d7d51cd308d, State: Initialized, Role: FOLLOWER
I20260619 18:50:04.716261 17546 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3fb469ed29c41ada7bf3d7d51cd308d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3fb469ed29c41ada7bf3d7d51cd308d" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 42939 } } peers { permanent_uuid: "f33d9a082430441494dbb1e660e29173" member_type: VOTER last_known_addr { host: "127.3.243.125" port: 32927 } } peers { permanent_uuid: "533331520ff242dfb22983d8aad9617c" member_type: VOTER last_known_addr { host: "127.3.243.124" port: 44199 } }
I20260619 18:50:04.720841 17624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:04.703696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56730 (local address 127.3.243.125:32927)
0619 18:50:04.704311 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0619 18:50:04.704315 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:50:04.704328 (+ 13us) server_negotiation.cc:407] Connection header received
0619 18:50:04.704362 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:50:04.704365 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:50:04.704414 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:50:04.704493 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:50:04.706013 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:04.707039 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:04.708768 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:04.709386 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:04.710097 (+ 711us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:04.714483 (+ 4386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:50:04.714488 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:50:04.714492 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:50:04.714541 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:50:04.717375 (+ 2834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:04.718243 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:04.718249 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:04.718253 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:04.718403 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:04.718942 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:04.718948 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:04.718952 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:04.719285 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:50:04.719576 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0619 18:50:04.720190 (+ 614us) server_negotiation.cc:299] Negotiation successful
0619 18:50:04.720506 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521}
I20260619 18:50:04.720995 17694 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 533331520ff242dfb22983d8aad9617c: Bootstrap starting.
I20260619 18:50:04.724825 17694 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 533331520ff242dfb22983d8aad9617c: Neither blocks nor log segments found. Creating new log.
I20260619 18:50:04.724924 17702 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3fb469ed29c41ada7bf3d7d51cd308d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3fb469ed29c41ada7bf3d7d51cd308d" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 42939 } } peers { permanent_uuid: "f33d9a082430441494dbb1e660e29173" member_type: VOTER last_known_addr { host: "127.3.243.125" port: 32927 } } peers { permanent_uuid: "533331520ff242dfb22983d8aad9617c" member_type: VOTER last_known_addr { host: "127.3.243.124" port: 44199 } } }
I20260619 18:50:04.725374 17702 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3fb469ed29c41ada7bf3d7d51cd308d [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:50:04.726100 17694 log.cc:824] T 00000000000000000000000000000000 P 533331520ff242dfb22983d8aad9617c: Log is configured to *not* fsync() on all Append() calls
I20260619 18:50:04.726486 17546 sys_catalog.cc:564] T 00000000000000000000000000000000 P b3fb469ed29c41ada7bf3d7d51cd308d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:50:04.730664 4045 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:50:04.736114 17694 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 533331520ff242dfb22983d8aad9617c: No bootstrap required, opened a new log
I20260619 18:50:04.744596 17694 raft_consensus.cc:348] T 00000000000000000000000000000000 P 533331520ff242dfb22983d8aad9617c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3fb469ed29c41ada7bf3d7d51cd308d" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 42939 } } peers { permanent_uuid: "f33d9a082430441494dbb1e660e29173" member_type: VOTER last_known_addr { host: "127.3.243.125" port: 32927 } } peers { permanent_uuid: "533331520ff242dfb22983d8aad9617c" member_type: VOTER last_known_addr { host: "127.3.243.124" port: 44199 } }
I20260619 18:50:04.745182 17694 raft_consensus.cc:374] T 00000000000000000000000000000000 P 533331520ff242dfb22983d8aad9617c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:50:04.745344 17694 raft_consensus.cc:729] T 00000000000000000000000000000000 P 533331520ff242dfb22983d8aad9617c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 533331520ff242dfb22983d8aad9617c, State: Initialized, Role: FOLLOWER
I20260619 18:50:04.746189 17694 consensus_queue.cc:260] T 00000000000000000000000000000000 P 533331520ff242dfb22983d8aad9617c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3fb469ed29c41ada7bf3d7d51cd308d" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 42939 } } peers { permanent_uuid: "f33d9a082430441494dbb1e660e29173" member_type: VOTER last_known_addr { host: "127.3.243.125" port: 32927 } } peers { permanent_uuid: "533331520ff242dfb22983d8aad9617c" member_type: VOTER last_known_addr { host: "127.3.243.124" port: 44199 } }
I20260619 18:50:04.762897 17550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:04.743004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47268 (local address 127.3.243.126:42939)
0619 18:50:04.743189 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 18:50:04.743196 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:50:04.744093 (+ 897us) server_negotiation.cc:407] Connection header received
0619 18:50:04.744402 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:50:04.744408 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:50:04.744493 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:50:04.744566 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:50:04.746951 (+ 2385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:04.748229 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:04.750378 (+ 2149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:04.751106 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:04.751206 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:04.755157 (+ 3951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:50:04.755164 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:50:04.755170 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:50:04.755235 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:50:04.759264 (+ 4029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:04.760275 (+ 1011us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:04.760283 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:04.760287 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:04.760503 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:04.761005 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:04.761010 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:04.761014 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:04.761367 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:50:04.761629 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0619 18:50:04.762210 (+ 581us) server_negotiation.cc:299] Negotiation successful
0619 18:50:04.762474 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 18:50:04.764374 17715 sys_catalog.cc:455] T 00000000000000000000000000000000 P 533331520ff242dfb22983d8aad9617c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3fb469ed29c41ada7bf3d7d51cd308d" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 42939 } } peers { permanent_uuid: "f33d9a082430441494dbb1e660e29173" member_type: VOTER last_known_addr { host: "127.3.243.125" port: 32927 } } peers { permanent_uuid: "533331520ff242dfb22983d8aad9617c" member_type: VOTER last_known_addr { host: "127.3.243.124" port: 44199 } } }
I20260619 18:50:04.764792 17715 sys_catalog.cc:458] T 00000000000000000000000000000000 P 533331520ff242dfb22983d8aad9617c [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:50:04.767166 17715 raft_consensus.cc:482] T 00000000000000000000000000000000 P 533331520ff242dfb22983d8aad9617c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:50:04.767519 17715 raft_consensus.cc:504] T 00000000000000000000000000000000 P 533331520ff242dfb22983d8aad9617c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3fb469ed29c41ada7bf3d7d51cd308d" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 42939 } } peers { permanent_uuid: "f33d9a082430441494dbb1e660e29173" member_type: VOTER last_known_addr { host: "127.3.243.125" port: 32927 } } peers { permanent_uuid: "533331520ff242dfb22983d8aad9617c" member_type: VOTER last_known_addr { host: "127.3.243.124" port: 44199 } }
I20260619 18:50:04.770010 17521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "533331520ff242dfb22983d8aad9617c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3fb469ed29c41ada7bf3d7d51cd308d" is_pre_election: true
I20260619 18:50:04.770613 17521 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b3fb469ed29c41ada7bf3d7d51cd308d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 533331520ff242dfb22983d8aad9617c in term 0.
I20260619 18:50:04.771327 17644 leader_election.cc:302] T 00000000000000000000000000000000 P 533331520ff242dfb22983d8aad9617c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 533331520ff242dfb22983d8aad9617c, b3fb469ed29c41ada7bf3d7d51cd308d; no voters:
I20260619 18:50:04.773123 17715 leader_election.cc:288] T 00000000000000000000000000000000 P 533331520ff242dfb22983d8aad9617c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3fb469ed29c41ada7bf3d7d51cd308d (127.3.243.126:42939), f33d9a082430441494dbb1e660e29173 (127.3.243.125:32927)
I20260619 18:50:04.776870 17624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:04.743737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56742 (local address 127.3.243.125:32927)
0619 18:50:04.743898 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 18:50:04.743903 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:50:04.759413 (+ 15510us) server_negotiation.cc:407] Connection header received
0619 18:50:04.759941 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:50:04.759948 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:50:04.760035 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:50:04.760104 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:50:04.761904 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:04.763024 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:04.765381 (+ 2357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:04.766088 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:04.766219 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:04.769855 (+ 3636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:50:04.769862 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:50:04.769867 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:50:04.769911 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:50:04.773481 (+ 3570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:04.774361 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:04.774367 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:04.774371 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:04.774533 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:04.774987 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:04.774993 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:04.774997 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:04.775379 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:50:04.775642 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0619 18:50:04.776212 (+ 570us) server_negotiation.cc:299] Negotiation successful
0619 18:50:04.776471 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 18:50:04.778254 17694 sys_catalog.cc:564] T 00000000000000000000000000000000 P 533331520ff242dfb22983d8aad9617c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:50:04.779464 17715 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 533331520ff242dfb22983d8aad9617c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:50:04.779624 17715 raft_consensus.cc:482] T 00000000000000000000000000000000 P 533331520ff242dfb22983d8aad9617c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:50:04.779726 17715 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 533331520ff242dfb22983d8aad9617c [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:50:04.782469 17699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:04.749871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51520 (local address 127.3.243.124:44199)
0619 18:50:04.750046 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 18:50:04.750051 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:50:04.763790 (+ 13739us) server_negotiation.cc:407] Connection header received
0619 18:50:04.765187 (+ 1397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:50:04.765192 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:50:04.765286 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:50:04.765353 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:50:04.766994 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:04.768036 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:04.770015 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:04.770603 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:04.770698 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:04.774035 (+ 3337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:50:04.774042 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:50:04.774048 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:50:04.774086 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:50:04.778111 (+ 4025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:04.779905 (+ 1794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:04.779912 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:04.779916 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:04.780130 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:04.780585 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:04.780590 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:04.780594 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:04.780930 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:50:04.781233 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0619 18:50:04.781904 (+ 671us) server_negotiation.cc:299] Negotiation successful
0619 18:50:04.782119 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 18:50:04.783074 17715 raft_consensus.cc:504] T 00000000000000000000000000000000 P 533331520ff242dfb22983d8aad9617c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3fb469ed29c41ada7bf3d7d51cd308d" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 42939 } } peers { permanent_uuid: "f33d9a082430441494dbb1e660e29173" member_type: VOTER last_known_addr { host: "127.3.243.125" port: 32927 } } peers { permanent_uuid: "533331520ff242dfb22983d8aad9617c" member_type: VOTER last_known_addr { host: "127.3.243.124" port: 44199 } }
I20260619 18:50:04.784313 17521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "533331520ff242dfb22983d8aad9617c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3fb469ed29c41ada7bf3d7d51cd308d"
I20260619 18:50:04.784539 17521 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b3fb469ed29c41ada7bf3d7d51cd308d [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:50:04.784929 17715 leader_election.cc:288] T 00000000000000000000000000000000 P 533331520ff242dfb22983d8aad9617c [CANDIDATE]: Term 1 election: Requested vote from peers b3fb469ed29c41ada7bf3d7d51cd308d (127.3.243.126:42939), f33d9a082430441494dbb1e660e29173 (127.3.243.125:32927)
I20260619 18:50:04.788371 17521 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b3fb469ed29c41ada7bf3d7d51cd308d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 533331520ff242dfb22983d8aad9617c in term 1.
I20260619 18:50:04.788859 17644 leader_election.cc:302] T 00000000000000000000000000000000 P 533331520ff242dfb22983d8aad9617c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 533331520ff242dfb22983d8aad9617c, b3fb469ed29c41ada7bf3d7d51cd308d; no voters:
I20260619 18:50:04.789287 17715 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 533331520ff242dfb22983d8aad9617c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:50:04.789453 17715 raft_consensus.cc:686] T 00000000000000000000000000000000 P 533331520ff242dfb22983d8aad9617c [term 1 LEADER]: Becoming Leader. State: Replica: 533331520ff242dfb22983d8aad9617c, State: Running, Role: LEADER
I20260619 18:50:04.790030 17715 consensus_queue.cc:237] T 00000000000000000000000000000000 P 533331520ff242dfb22983d8aad9617c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3fb469ed29c41ada7bf3d7d51cd308d" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 42939 } } peers { permanent_uuid: "f33d9a082430441494dbb1e660e29173" member_type: VOTER last_known_addr { host: "127.3.243.125" port: 32927 } } peers { permanent_uuid: "533331520ff242dfb22983d8aad9617c" member_type: VOTER last_known_addr { host: "127.3.243.124" port: 44199 } }
I20260619 18:50:04.794098 17713 sys_catalog.cc:455] T 00000000000000000000000000000000 P 533331520ff242dfb22983d8aad9617c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 533331520ff242dfb22983d8aad9617c. Latest consensus state: current_term: 1 leader_uuid: "533331520ff242dfb22983d8aad9617c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3fb469ed29c41ada7bf3d7d51cd308d" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 42939 } } peers { permanent_uuid: "f33d9a082430441494dbb1e660e29173" member_type: VOTER last_known_addr { host: "127.3.243.125" port: 32927 } } peers { permanent_uuid: "533331520ff242dfb22983d8aad9617c" member_type: VOTER last_known_addr { host: "127.3.243.124" port: 44199 } } }
I20260619 18:50:04.794543 17713 sys_catalog.cc:458] T 00000000000000000000000000000000 P 533331520ff242dfb22983d8aad9617c [sys.catalog]: This master's current role is: LEADER
I20260619 18:50:04.803653 17723 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:50:04.809980 17723 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:50:04.824182 17521 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b3fb469ed29c41ada7bf3d7d51cd308d [term 1 FOLLOWER]: Refusing update from remote peer 533331520ff242dfb22983d8aad9617c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:50:04.825256 17713 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 533331520ff242dfb22983d8aad9617c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3fb469ed29c41ada7bf3d7d51cd308d" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:50:04.841571 17702 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3fb469ed29c41ada7bf3d7d51cd308d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 533331520ff242dfb22983d8aad9617c. Latest consensus state: current_term: 1 leader_uuid: "533331520ff242dfb22983d8aad9617c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3fb469ed29c41ada7bf3d7d51cd308d" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 42939 } } peers { permanent_uuid: "f33d9a082430441494dbb1e660e29173" member_type: VOTER last_known_addr { host: "127.3.243.125" port: 32927 } } peers { permanent_uuid: "533331520ff242dfb22983d8aad9617c" member_type: VOTER last_known_addr { host: "127.3.243.124" port: 44199 } } }
I20260619 18:50:04.841918 17702 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3fb469ed29c41ada7bf3d7d51cd308d [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:50:04.846545 17479 master_runner.cc:186] Error getting master registration for 127.3.243.125:32927: OK, instance_id { permanent_uuid: "f33d9a082430441494dbb1e660e29173" instance_seqno: 1781895004169757 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 18:50:04.859549 17479 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260619 18:50:04.865931 17736 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b3fb469ed29c41ada7bf3d7d51cd308d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 18:50:04.866065 17736 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 18:50:04.887751 17620 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.243.124" port: 44199 } attempt: 5
I20260619 18:50:04.891112 17702 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3fb469ed29c41ada7bf3d7d51cd308d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "533331520ff242dfb22983d8aad9617c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3fb469ed29c41ada7bf3d7d51cd308d" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 42939 } } peers { permanent_uuid: "f33d9a082430441494dbb1e660e29173" member_type: VOTER last_known_addr { host: "127.3.243.125" port: 32927 } } peers { permanent_uuid: "533331520ff242dfb22983d8aad9617c" member_type: VOTER last_known_addr { host: "127.3.243.124" port: 44199 } } }
I20260619 18:50:04.891366 17702 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3fb469ed29c41ada7bf3d7d51cd308d [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:50:04.895696 17715 sys_catalog.cc:455] T 00000000000000000000000000000000 P 533331520ff242dfb22983d8aad9617c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "533331520ff242dfb22983d8aad9617c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3fb469ed29c41ada7bf3d7d51cd308d" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 42939 } } peers { permanent_uuid: "f33d9a082430441494dbb1e660e29173" member_type: VOTER last_known_addr { host: "127.3.243.125" port: 32927 } } peers { permanent_uuid: "533331520ff242dfb22983d8aad9617c" member_type: VOTER last_known_addr { host: "127.3.243.124" port: 44199 } } }
I20260619 18:50:04.896013 17715 sys_catalog.cc:458] T 00000000000000000000000000000000 P 533331520ff242dfb22983d8aad9617c [sys.catalog]: This master's current role is: LEADER
Jun 19 18:50:04 dist-test-slave-m9wg krb5kdc[17466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895004, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.125@KRBTEST.COM for kudu/127.3.243.124@KRBTEST.COM
I20260619 18:50:04.898849 17723 catalog_manager.cc:1347] Generated new cluster ID: 249ea48abd4f4dd3a5a7df437ed54c3a
I20260619 18:50:04.898962 17723 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:50:04.905049 17622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:04.888428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.124:44199 (local address 127.0.0.1:51530)
0619 18:50:04.888814 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:04.888834 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 18:50:04.888948 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:50:04.889819 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:50:04.889824 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:50:04.890359 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:50:04.890568 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:50:04.890582 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:50:04.892099 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:50:04.892105 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:50:04.893539 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:50:04.893554 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:50:04.894756 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:50:04.894762 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:50:04.895031 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:04.895683 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:50:04.895707 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:50:04.898486 (+ 2779us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:50:04.902498 (+ 4012us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:50:04.902503 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:50:04.902507 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:50:04.902970 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:50:04.903371 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:50:04.903376 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:50:04.903379 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:50:04.903522 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:50:04.904132 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:50:04.904137 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:50:04.904296 (+ 159us) client_negotiation.cc:769] Sending connection context
0619 18:50:04.904428 (+ 132us) client_negotiation.cc:240] Negotiation successful
0619 18:50:04.904683 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258}
I20260619 18:50:04.905905 17699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:04.889177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51530 (local address 127.3.243.124:44199)
0619 18:50:04.889356 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 18:50:04.889362 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:50:04.889380 (+ 18us) server_negotiation.cc:407] Connection header received
0619 18:50:04.889437 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:50:04.889443 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:50:04.889517 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:50:04.889613 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:50:04.890744 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:04.891925 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:04.893708 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:04.894442 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:04.894561 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:04.898697 (+ 4136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:50:04.898704 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:50:04.898708 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:50:04.898745 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:50:04.902347 (+ 3602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:04.903100 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:04.903106 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:04.903110 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:04.903248 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:04.903630 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:04.903635 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:04.903638 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:04.903948 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:50:04.904080 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0619 18:50:04.905376 (+ 1296us) server_negotiation.cc:299] Negotiation successful
0619 18:50:04.905578 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 18:50:04.911432 17620 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f33d9a082430441494dbb1e660e29173: Bootstrap starting.
I20260619 18:50:04.915503 17620 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f33d9a082430441494dbb1e660e29173: Neither blocks nor log segments found. Creating new log.
I20260619 18:50:04.916857 17620 log.cc:824] T 00000000000000000000000000000000 P f33d9a082430441494dbb1e660e29173: Log is configured to *not* fsync() on all Append() calls
I20260619 18:50:04.920746 17620 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f33d9a082430441494dbb1e660e29173: No bootstrap required, opened a new log
I20260619 18:50:04.930759 17723 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:50:04.932107 17620 raft_consensus.cc:348] T 00000000000000000000000000000000 P f33d9a082430441494dbb1e660e29173 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3fb469ed29c41ada7bf3d7d51cd308d" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 42939 } } peers { permanent_uuid: "f33d9a082430441494dbb1e660e29173" member_type: VOTER last_known_addr { host: "127.3.243.125" port: 32927 } } peers { permanent_uuid: "533331520ff242dfb22983d8aad9617c" member_type: VOTER last_known_addr { host: "127.3.243.124" port: 44199 } }
I20260619 18:50:04.932647 17620 raft_consensus.cc:374] T 00000000000000000000000000000000 P f33d9a082430441494dbb1e660e29173 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:50:04.932809 17620 raft_consensus.cc:729] T 00000000000000000000000000000000 P f33d9a082430441494dbb1e660e29173 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f33d9a082430441494dbb1e660e29173, State: Initialized, Role: FOLLOWER
I20260619 18:50:04.933646 17620 consensus_queue.cc:260] T 00000000000000000000000000000000 P f33d9a082430441494dbb1e660e29173 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3fb469ed29c41ada7bf3d7d51cd308d" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 42939 } } peers { permanent_uuid: "f33d9a082430441494dbb1e660e29173" member_type: VOTER last_known_addr { host: "127.3.243.125" port: 32927 } } peers { permanent_uuid: "533331520ff242dfb22983d8aad9617c" member_type: VOTER last_known_addr { host: "127.3.243.124" port: 44199 } }
I20260619 18:50:04.935294 17723 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:50:04.941624 17745 sys_catalog.cc:455] T 00000000000000000000000000000000 P f33d9a082430441494dbb1e660e29173 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3fb469ed29c41ada7bf3d7d51cd308d" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 42939 } } peers { permanent_uuid: "f33d9a082430441494dbb1e660e29173" member_type: VOTER last_known_addr { host: "127.3.243.125" port: 32927 } } peers { permanent_uuid: "533331520ff242dfb22983d8aad9617c" member_type: VOTER last_known_addr { host: "127.3.243.124" port: 44199 } } }
I20260619 18:50:04.944137 17594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "533331520ff242dfb22983d8aad9617c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f33d9a082430441494dbb1e660e29173"
I20260619 18:50:04.944546 17593 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f33d9a082430441494dbb1e660e29173 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:50:04.948081 17593 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f33d9a082430441494dbb1e660e29173 [term 1 FOLLOWER]: Refusing update from remote peer 533331520ff242dfb22983d8aad9617c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:50:04.948570 17595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "533331520ff242dfb22983d8aad9617c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f33d9a082430441494dbb1e660e29173" is_pre_election: true
I20260619 18:50:04.948951 17595 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f33d9a082430441494dbb1e660e29173 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 533331520ff242dfb22983d8aad9617c in term 1.
I20260619 18:50:04.949282 17742 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 533331520ff242dfb22983d8aad9617c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f33d9a082430441494dbb1e660e29173" member_type: VOTER last_known_addr { host: "127.3.243.125" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:50:04.944468 17620 sys_catalog.cc:564] T 00000000000000000000000000000000 P f33d9a082430441494dbb1e660e29173 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:50:04.964854 17723 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 533331520ff242dfb22983d8aad9617c: Generated new TSK 0
I20260619 18:50:04.966001 17723 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:50:04.946415 17627 master_runner.cc:186] Error getting master registration for 127.3.243.125:32927: OK, instance_id { permanent_uuid: "f33d9a082430441494dbb1e660e29173" instance_seqno: 1781895004169757 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 18:50:04.969039 17745 sys_catalog.cc:458] T 00000000000000000000000000000000 P f33d9a082430441494dbb1e660e29173 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:50:04.974952 17627 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 18:50:04.991726 17758 mvcc.cc:205] Tried to move back new op lower bound from 7298641939742613504 to 7298641939625189376. Current Snapshot: MvccSnapshot[applied={T|T < 7298641939742613504}]
I20260619 18:50:04.997138 17745 sys_catalog.cc:455] T 00000000000000000000000000000000 P f33d9a082430441494dbb1e660e29173 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 533331520ff242dfb22983d8aad9617c. Latest consensus state: current_term: 1 leader_uuid: "533331520ff242dfb22983d8aad9617c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3fb469ed29c41ada7bf3d7d51cd308d" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 42939 } } peers { permanent_uuid: "f33d9a082430441494dbb1e660e29173" member_type: VOTER last_known_addr { host: "127.3.243.125" port: 32927 } } peers { permanent_uuid: "533331520ff242dfb22983d8aad9617c" member_type: VOTER last_known_addr { host: "127.3.243.124" port: 44199 } } }
I20260619 18:50:04.997467 17745 sys_catalog.cc:458] T 00000000000000000000000000000000 P f33d9a082430441494dbb1e660e29173 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:50:04.997347 17715 sys_catalog.cc:455] T 00000000000000000000000000000000 P 533331520ff242dfb22983d8aad9617c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "533331520ff242dfb22983d8aad9617c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3fb469ed29c41ada7bf3d7d51cd308d" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 42939 } } peers { permanent_uuid: "f33d9a082430441494dbb1e660e29173" member_type: VOTER last_known_addr { host: "127.3.243.125" port: 32927 } } peers { permanent_uuid: "533331520ff242dfb22983d8aad9617c" member_type: VOTER last_known_addr { host: "127.3.243.124" port: 44199 } } }
I20260619 18:50:04.997622 17715 sys_catalog.cc:458] T 00000000000000000000000000000000 P 533331520ff242dfb22983d8aad9617c [sys.catalog]: This master's current role is: LEADER
I20260619 18:50:04.997797 17747 sys_catalog.cc:455] T 00000000000000000000000000000000 P f33d9a082430441494dbb1e660e29173 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "533331520ff242dfb22983d8aad9617c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3fb469ed29c41ada7bf3d7d51cd308d" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 42939 } } peers { permanent_uuid: "f33d9a082430441494dbb1e660e29173" member_type: VOTER last_known_addr { host: "127.3.243.125" port: 32927 } } peers { permanent_uuid: "533331520ff242dfb22983d8aad9617c" member_type: VOTER last_known_addr { host: "127.3.243.124" port: 44199 } } }
I20260619 18:50:04.998175 17747 sys_catalog.cc:458] T 00000000000000000000000000000000 P f33d9a082430441494dbb1e660e29173 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:50:05.027159 17762 catalog_manager.cc:1259] Loaded cluster ID: 249ea48abd4f4dd3a5a7df437ed54c3a
I20260619 18:50:05.027276 17762 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f33d9a082430441494dbb1e660e29173: loading cluster ID for follower catalog manager: success
I20260619 18:50:05.030874 17762 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f33d9a082430441494dbb1e660e29173: acquiring CA information for follower catalog manager: success
I20260619 18:50:05.033396 17762 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f33d9a082430441494dbb1e660e29173: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 18:50:05.867964 17736 catalog_manager.cc:1259] Loaded cluster ID: 249ea48abd4f4dd3a5a7df437ed54c3a
I20260619 18:50:05.868067 17736 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b3fb469ed29c41ada7bf3d7d51cd308d: loading cluster ID for follower catalog manager: success
I20260619 18:50:05.870590 17736 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b3fb469ed29c41ada7bf3d7d51cd308d: acquiring CA information for follower catalog manager: success
I20260619 18:50:05.872581 17736 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b3fb469ed29c41ada7bf3d7d51cd308d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 18:50:06.107511 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 17479
I20260619 18:50:06.108655 17541 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:50:06.216522 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 17479
I20260619 18:50:06.233211 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 17553
I20260619 18:50:06.234270 17615 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:50:06.338481 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 17553
I20260619 18:50:06.355012 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 17627
I20260619 18:50:06.356065 17689 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:50:06.462952 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 17627
I20260619 18:50:06.463035 17772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:06.462268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:42939 (local address 127.0.0.1:47282)
0619 18:50:06.462663 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:06.462820 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:42939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":295,"threads_started":1}
2026-06-19T18:50:06Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4368 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781894810715282-4045-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:50:06 dist-test-slave-m9wg krb5kdc[17775](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:50:06 dist-test-slave-m9wg krb5kdc[17775](info): set up 2 sockets
Jun 19 18:50:06 dist-test-slave-m9wg krb5kdc[17775](info): commencing operation
krb5kdc: starting...
W20260619 18:50:07.484390 4045 mini_kdc.cc:121] Time spent starting KDC: real 0.985s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781894810715282-4045-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781894810715282-4045-0/krb5kdc/test-user.keytab.
Jun 19 18:50:07 dist-test-slave-m9wg krb5kdc[17775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895007, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:50:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:50:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.243.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab.
Entry for principal kudu/127.3.243.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab.
I20260619 18:50:07.685582 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781894810715282-4045-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781894810715282-4045-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781894810715282-4045-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.126:33859
--webserver_interface=127.3.243.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab
--principal=kudu/127.3.243.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:38789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.126:33859
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:50:07.995256 17788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:50:07.995530 17788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:50:07.995756 17788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:50:08.002928 17788 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:50:08.003027 17788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:50:08.003103 17788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:50:08.003177 17788 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:50:08.003234 17788 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:50:08.012871 17788 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:38789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781894810715282-4045-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781894810715282-4045-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.243.126:33859
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab
--principal=kudu/127.3.243.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.126:33859
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.243.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781894810715282-4045-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:50:08.014130 17788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:50:08.015414 17788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:50:08.031497 17794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:50:08.031587 17796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:50:08.031562 17793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:50:08.032454 17788 server_base.cc:1034] running on GCE node
I20260619 18:50:08.033147 17788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:50:08.034475 17788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:50:08.035666 17788 hybrid_clock.cc:648] HybridClock initialized: now 1781895008035610 us; error 47 us; skew 500 ppm
I20260619 18:50:08.035982 17788 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:50:08 dist-test-slave-m9wg krb5kdc[17775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895008, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:50:08.044422 17788 init.cc:377] Logged in from keytab as kudu/127.3.243.126@KRBTEST.COM (short username kudu)
I20260619 18:50:08.045804 17788 webserver.cc:460] Webserver started at http://127.3.243.126:36001/ using document root <none> and password file <none>
I20260619 18:50:08.046449 17788 fs_manager.cc:359] Metadata directory not provided
I20260619 18:50:08.046572 17788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:50:08.046867 17788 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:50:08.049767 17788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781894810715282-4045-0/minicluster-data/master-0/data/instance:
uuid: "23555dce316d4b22bb775a7e88d73a09"
format_stamp: "Formatted at 2026-06-19 18:50:08 on dist-test-slave-m9wg"
server_key: "57b6c531ac39b27f925e6085dd22c702"
server_key_iv: "2aa1279c0135c5e1060d630c17524696"
server_key_version: "encryptionkey@0"
I20260619 18:50:08.050457 17788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781894810715282-4045-0/minicluster-data/master-0/wal/instance:
uuid: "23555dce316d4b22bb775a7e88d73a09"
format_stamp: "Formatted at 2026-06-19 18:50:08 on dist-test-slave-m9wg"
server_key: "57b6c531ac39b27f925e6085dd22c702"
server_key_iv: "2aa1279c0135c5e1060d630c17524696"
server_key_version: "encryptionkey@0"
I20260619 18:50:08.055418 17788 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 18:50:08.058807 17803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:50:08.060012 17788 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260619 18:50:08.060201 17788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781894810715282-4045-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781894810715282-4045-0/minicluster-data/master-0/wal
uuid: "23555dce316d4b22bb775a7e88d73a09"
format_stamp: "Formatted at 2026-06-19 18:50:08 on dist-test-slave-m9wg"
server_key: "57b6c531ac39b27f925e6085dd22c702"
server_key_iv: "2aa1279c0135c5e1060d630c17524696"
server_key_version: "encryptionkey@0"
I20260619 18:50:08.060374 17788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781894810715282-4045-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781894810715282-4045-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781894810715282-4045-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:50:08.074852 17788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:50:08.078409 17788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:50:08.078655 17788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:50:08.095252 17788 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.126:33859
I20260619 18:50:08.095280 17854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.126:33859 every 8 connection(s)
I20260619 18:50:08.096592 17788 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
I20260619 18:50:08.100348 17855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:50:08.102698 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 17788
I20260619 18:50:08.102898 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781894810715282-4045-0/minicluster-data/master-0/wal/instance
I20260619 18:50:08.103513 4045 external_mini_cluster.cc:1428] Setting key 7d9cef1b86139855b8744aaff708ed28
I20260619 18:50:08.112524 17855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23555dce316d4b22bb775a7e88d73a09: Bootstrap starting.
I20260619 18:50:08.115909 17855 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 23555dce316d4b22bb775a7e88d73a09: Neither blocks nor log segments found. Creating new log.
I20260619 18:50:08.116884 17855 log.cc:824] T 00000000000000000000000000000000 P 23555dce316d4b22bb775a7e88d73a09: Log is configured to *not* fsync() on all Append() calls
Jun 19 18:50:08 dist-test-slave-m9wg krb5kdc[17775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895007, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.126@KRBTEST.COM
I20260619 18:50:08.119764 17855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23555dce316d4b22bb775a7e88d73a09: No bootstrap required, opened a new log
I20260619 18:50:08.125943 17858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:08.105278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38798 (local address 127.3.243.126:33859)
0619 18:50:08.106059 (+ 781us) server_negotiation.cc:206] Beginning negotiation
0619 18:50:08.106074 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 18:50:08.106124 (+ 50us) server_negotiation.cc:407] Connection header received
0619 18:50:08.108389 (+ 2265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:50:08.108418 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:50:08.108906 (+ 488us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:50:08.109344 (+ 438us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:50:08.110622 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:08.112533 (+ 1911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:08.113784 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:08.114793 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:08.114989 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:08.119171 (+ 4182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:50:08.119179 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:50:08.119193 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:50:08.119227 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:50:08.122145 (+ 2918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:08.122975 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:08.122981 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:08.122987 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:08.123152 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:08.123525 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:08.123528 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:08.123530 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:08.123987 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:50:08.124267 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0619 18:50:08.124650 (+ 383us) server_negotiation.cc:299] Negotiation successful
0619 18:50:08.124944 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":323,"threads_started":1}
I20260619 18:50:08.128398 17855 raft_consensus.cc:348] T 00000000000000000000000000000000 P 23555dce316d4b22bb775a7e88d73a09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23555dce316d4b22bb775a7e88d73a09" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 33859 } }
I20260619 18:50:08.128813 17855 raft_consensus.cc:374] T 00000000000000000000000000000000 P 23555dce316d4b22bb775a7e88d73a09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:50:08.128919 17855 raft_consensus.cc:729] T 00000000000000000000000000000000 P 23555dce316d4b22bb775a7e88d73a09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23555dce316d4b22bb775a7e88d73a09, State: Initialized, Role: FOLLOWER
I20260619 18:50:08.129511 17855 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23555dce316d4b22bb775a7e88d73a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23555dce316d4b22bb775a7e88d73a09" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 33859 } }
I20260619 18:50:08.129714 4045 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:50:08.129781 17855 raft_consensus.cc:388] T 00000000000000000000000000000000 P 23555dce316d4b22bb775a7e88d73a09 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:50:08.129907 17855 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23555dce316d4b22bb775a7e88d73a09 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:50:08.130051 17855 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 23555dce316d4b22bb775a7e88d73a09 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:50:08.132941 17855 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23555dce316d4b22bb775a7e88d73a09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23555dce316d4b22bb775a7e88d73a09" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 33859 } }
I20260619 18:50:08.133448 17855 leader_election.cc:302] T 00000000000000000000000000000000 P 23555dce316d4b22bb775a7e88d73a09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 23555dce316d4b22bb775a7e88d73a09; no voters:
I20260619 18:50:08.133956 17855 leader_election.cc:288] T 00000000000000000000000000000000 P 23555dce316d4b22bb775a7e88d73a09 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:50:08.134061 17860 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 23555dce316d4b22bb775a7e88d73a09 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:50:08.134651 17860 raft_consensus.cc:686] T 00000000000000000000000000000000 P 23555dce316d4b22bb775a7e88d73a09 [term 1 LEADER]: Becoming Leader. State: Replica: 23555dce316d4b22bb775a7e88d73a09, State: Running, Role: LEADER
I20260619 18:50:08.135139 17860 consensus_queue.cc:237] T 00000000000000000000000000000000 P 23555dce316d4b22bb775a7e88d73a09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23555dce316d4b22bb775a7e88d73a09" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 33859 } }
I20260619 18:50:08.135741 17855 sys_catalog.cc:564] T 00000000000000000000000000000000 P 23555dce316d4b22bb775a7e88d73a09 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:50:08.137590 17861 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23555dce316d4b22bb775a7e88d73a09 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "23555dce316d4b22bb775a7e88d73a09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23555dce316d4b22bb775a7e88d73a09" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 33859 } } }
I20260619 18:50:08.137535 17862 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23555dce316d4b22bb775a7e88d73a09 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23555dce316d4b22bb775a7e88d73a09. Latest consensus state: current_term: 1 leader_uuid: "23555dce316d4b22bb775a7e88d73a09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23555dce316d4b22bb775a7e88d73a09" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 33859 } } }
I20260619 18:50:08.137928 17861 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23555dce316d4b22bb775a7e88d73a09 [sys.catalog]: This master's current role is: LEADER
I20260619 18:50:08.138049 17862 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23555dce316d4b22bb775a7e88d73a09 [sys.catalog]: This master's current role is: LEADER
I20260619 18:50:08.139391 17867 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:50:08.144214 17867 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:50:08.152549 17867 catalog_manager.cc:1347] Generated new cluster ID: 058f83ad914942dab77699a59f4d4985
I20260619 18:50:08.152642 17867 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:50:08.181339 17867 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:50:08.183218 17867 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:50:08.195199 17867 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 23555dce316d4b22bb775a7e88d73a09: Generated new TSK 0
I20260619 18:50:08.196182 17867 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:50:08.241814 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 17788
I20260619 18:50:08.242846 17850 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:50:08.337415 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 17788
2026-06-19T18:50:08Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1870 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:50:08 dist-test-slave-m9wg krb5kdc[17883](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:50:08 dist-test-slave-m9wg krb5kdc[17883](info): set up 2 sockets
Jun 19 18:50:08 dist-test-slave-m9wg krb5kdc[17883](info): commencing operation
krb5kdc: starting...
W20260619 18:50:09.336529 4045 mini_kdc.cc:121] Time spent starting KDC: real 0.966s user 0.001s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/krb5kdc/test-user.keytab.
Jun 19 18:50:09 dist-test-slave-m9wg krb5kdc[17883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895009, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:50:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:50:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.243.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab.
Entry for principal kudu/127.3.243.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab.
I20260619 18:50:09.550748 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.126:39697
--webserver_interface=127.3.243.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab
--principal=kudu/127.3.243.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:46639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.126:39697
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:50:09.859647 17896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:50:09.859951 17896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:50:09.860095 17896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:50:09.867362 17896 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:50:09.867482 17896 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:50:09.867560 17896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:50:09.867640 17896 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:50:09.867699 17896 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:50:09.877480 17896 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:46639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.243.126:39697
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab
--principal=kudu/127.3.243.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.126:39697
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.243.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:50:09.878805 17896 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:50:09.880093 17896 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:50:09.891078 17901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:50:09.891161 17902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:50:09.891464 17904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:50:09.891904 17896 server_base.cc:1034] running on GCE node
I20260619 18:50:09.892633 17896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:50:09.893970 17896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:50:09.895164 17896 hybrid_clock.cc:648] HybridClock initialized: now 1781895009895117 us; error 35 us; skew 500 ppm
I20260619 18:50:09.895478 17896 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:50:09 dist-test-slave-m9wg krb5kdc[17883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895009, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:50:09.898424 17896 init.cc:377] Logged in from keytab as kudu/127.3.243.126@KRBTEST.COM (short username kudu)
I20260619 18:50:09.899991 17896 webserver.cc:460] Webserver started at http://127.3.243.126:39799/ using document root <none> and password file <none>
I20260619 18:50:09.900689 17896 fs_manager.cc:359] Metadata directory not provided
I20260619 18:50:09.900807 17896 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:50:09.901103 17896 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:50:09.903999 17896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/master-0/data/instance:
uuid: "22ae91d05d6a4613906a3966068df987"
format_stamp: "Formatted at 2026-06-19 18:50:09 on dist-test-slave-m9wg"
server_key: "d2bfefb082a12cc1c57cf924a95d3392"
server_key_iv: "e8a328116c7198cab94c0c4d14d33ba9"
server_key_version: "encryptionkey@0"
I20260619 18:50:09.904791 17896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/master-0/wal/instance:
uuid: "22ae91d05d6a4613906a3966068df987"
format_stamp: "Formatted at 2026-06-19 18:50:09 on dist-test-slave-m9wg"
server_key: "d2bfefb082a12cc1c57cf924a95d3392"
server_key_iv: "e8a328116c7198cab94c0c4d14d33ba9"
server_key_version: "encryptionkey@0"
I20260619 18:50:09.909677 17896 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 18:50:09.912806 17911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:50:09.913980 17896 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 18:50:09.914131 17896 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/master-0/wal
uuid: "22ae91d05d6a4613906a3966068df987"
format_stamp: "Formatted at 2026-06-19 18:50:09 on dist-test-slave-m9wg"
server_key: "d2bfefb082a12cc1c57cf924a95d3392"
server_key_iv: "e8a328116c7198cab94c0c4d14d33ba9"
server_key_version: "encryptionkey@0"
I20260619 18:50:09.914292 17896 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:50:09.938133 17896 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:50:09.941906 17896 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:50:09.942153 17896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:50:09.958387 17896 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.126:39697
I20260619 18:50:09.958421 17962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.126:39697 every 8 connection(s)
I20260619 18:50:09.959614 17896 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
I20260619 18:50:09.960170 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 17896
I20260619 18:50:09.960413 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/master-0/wal/instance
I20260619 18:50:09.961129 4045 external_mini_cluster.cc:1428] Setting key f895c59aa88b06ebef56d30e837719b8
I20260619 18:50:09.963739 17963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 18:50:09 dist-test-slave-m9wg krb5kdc[17883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895009, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.126@KRBTEST.COM
I20260619 18:50:09.975890 17963 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22ae91d05d6a4613906a3966068df987: Bootstrap starting.
I20260619 18:50:09.979521 17963 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 22ae91d05d6a4613906a3966068df987: Neither blocks nor log segments found. Creating new log.
I20260619 18:50:09.980736 17963 log.cc:824] T 00000000000000000000000000000000 P 22ae91d05d6a4613906a3966068df987: Log is configured to *not* fsync() on all Append() calls
I20260619 18:50:09.983879 17963 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22ae91d05d6a4613906a3966068df987: No bootstrap required, opened a new log
I20260619 18:50:09.984426 17966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:09.962946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54838 (local address 127.3.243.126:39697)
0619 18:50:09.963710 (+ 764us) server_negotiation.cc:206] Beginning negotiation
0619 18:50:09.963728 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 18:50:09.963777 (+ 49us) server_negotiation.cc:407] Connection header received
0619 18:50:09.966058 (+ 2281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:50:09.966084 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:50:09.966563 (+ 479us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:50:09.967000 (+ 437us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:50:09.968222 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:09.970193 (+ 1971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:09.971484 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:09.972508 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:09.972688 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:09.976581 (+ 3893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:50:09.976590 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:50:09.976607 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:50:09.976652 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:50:09.980328 (+ 3676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:09.980968 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:09.980976 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:09.980984 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:09.981172 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:09.981550 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:09.981553 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:09.981555 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:09.982097 (+ 542us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:50:09.982373 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0619 18:50:09.982716 (+ 343us) server_negotiation.cc:299] Negotiation successful
0619 18:50:09.983121 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":303,"threads_started":1}
I20260619 18:50:09.991361 17963 raft_consensus.cc:348] T 00000000000000000000000000000000 P 22ae91d05d6a4613906a3966068df987 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22ae91d05d6a4613906a3966068df987" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 39697 } }
I20260619 18:50:09.991701 17963 raft_consensus.cc:374] T 00000000000000000000000000000000 P 22ae91d05d6a4613906a3966068df987 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:50:09.991771 17963 raft_consensus.cc:729] T 00000000000000000000000000000000 P 22ae91d05d6a4613906a3966068df987 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22ae91d05d6a4613906a3966068df987, State: Initialized, Role: FOLLOWER
I20260619 18:50:09.992285 17963 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22ae91d05d6a4613906a3966068df987 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22ae91d05d6a4613906a3966068df987" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 39697 } }
I20260619 18:50:09.992497 17963 raft_consensus.cc:388] T 00000000000000000000000000000000 P 22ae91d05d6a4613906a3966068df987 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:50:09.992588 17963 raft_consensus.cc:482] T 00000000000000000000000000000000 P 22ae91d05d6a4613906a3966068df987 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:50:09.992714 17963 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 22ae91d05d6a4613906a3966068df987 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:50:09.994900 17963 raft_consensus.cc:504] T 00000000000000000000000000000000 P 22ae91d05d6a4613906a3966068df987 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22ae91d05d6a4613906a3966068df987" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 39697 } }
I20260619 18:50:09.995301 17963 leader_election.cc:302] T 00000000000000000000000000000000 P 22ae91d05d6a4613906a3966068df987 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22ae91d05d6a4613906a3966068df987; no voters:
I20260619 18:50:09.995791 17963 leader_election.cc:288] T 00000000000000000000000000000000 P 22ae91d05d6a4613906a3966068df987 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:50:09.995908 17968 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 22ae91d05d6a4613906a3966068df987 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:50:09.996511 17968 raft_consensus.cc:686] T 00000000000000000000000000000000 P 22ae91d05d6a4613906a3966068df987 [term 1 LEADER]: Becoming Leader. State: Replica: 22ae91d05d6a4613906a3966068df987, State: Running, Role: LEADER
I20260619 18:50:09.996995 17968 consensus_queue.cc:237] T 00000000000000000000000000000000 P 22ae91d05d6a4613906a3966068df987 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22ae91d05d6a4613906a3966068df987" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 39697 } }
I20260619 18:50:09.997601 17963 sys_catalog.cc:564] T 00000000000000000000000000000000 P 22ae91d05d6a4613906a3966068df987 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:50:09.999539 17970 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22ae91d05d6a4613906a3966068df987 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 22ae91d05d6a4613906a3966068df987. Latest consensus state: current_term: 1 leader_uuid: "22ae91d05d6a4613906a3966068df987" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22ae91d05d6a4613906a3966068df987" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 39697 } } }
I20260619 18:50:09.999872 17970 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22ae91d05d6a4613906a3966068df987 [sys.catalog]: This master's current role is: LEADER
I20260619 18:50:10.000268 17969 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22ae91d05d6a4613906a3966068df987 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "22ae91d05d6a4613906a3966068df987" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22ae91d05d6a4613906a3966068df987" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 39697 } } }
I20260619 18:50:10.000558 17969 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22ae91d05d6a4613906a3966068df987 [sys.catalog]: This master's current role is: LEADER
I20260619 18:50:10.001929 17977 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:50:10.006937 17977 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:50:10.015514 17977 catalog_manager.cc:1347] Generated new cluster ID: 13dc07384b5e42a49cc7067630c5ac3e
I20260619 18:50:10.015602 17977 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:50:10.027400 17977 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:50:10.029155 17977 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:50:10.036772 17977 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 22ae91d05d6a4613906a3966068df987: Generated new TSK 0
I20260619 18:50:10.037734 17977 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.243.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.65.keytab.
Entry for principal kudu/127.3.243.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.65.keytab.
I20260619 18:50:10.097640 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.65:0
--local_ip_for_outbound_sockets=127.3.243.65
--webserver_interface=127.3.243.65
--webserver_port=0
--tserver_master_addrs=127.3.243.126:39697
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.65.keytab
--principal=kudu/127.3.243.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:46639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:50:10.405985 17989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:50:10.406242 17989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:50:10.406371 17989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:50:10.413465 17989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:50:10.413568 17989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:50:10.413714 17989 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.65
I20260619 18:50:10.423244 17989 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:46639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.65.keytab
--principal=kudu/127.3.243.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.243.65
--webserver_port=0
--tserver_master_addrs=127.3.243.126:39697
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.65
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:50:10.424458 17989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:50:10.425727 17989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:50:10.437907 17994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:50:10.437935 17997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:50:10.437983 17995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:50:10.439041 17989 server_base.cc:1034] running on GCE node
I20260619 18:50:10.439589 17989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:50:10.440477 17989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:50:10.441672 17989 hybrid_clock.cc:648] HybridClock initialized: now 1781895010441624 us; error 39 us; skew 500 ppm
I20260619 18:50:10.441957 17989 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:50:10 dist-test-slave-m9wg krb5kdc[17883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895010, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:50:10.444844 17989 init.cc:377] Logged in from keytab as kudu/127.3.243.65@KRBTEST.COM (short username kudu)
I20260619 18:50:10.446172 17989 webserver.cc:460] Webserver started at http://127.3.243.65:34531/ using document root <none> and password file <none>
I20260619 18:50:10.446858 17989 fs_manager.cc:359] Metadata directory not provided
I20260619 18:50:10.446969 17989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:50:10.447227 17989 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:50:10.449951 17989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-0/data/instance:
uuid: "8d3f261588f14e158cc4b8a3388f5136"
format_stamp: "Formatted at 2026-06-19 18:50:10 on dist-test-slave-m9wg"
server_key: "4512017085bc64da101ab267e0829c0a"
server_key_iv: "f9aa9a356b97e4913808d79ae42c36f9"
server_key_version: "encryptionkey@0"
I20260619 18:50:10.450647 17989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-0/wal/instance:
uuid: "8d3f261588f14e158cc4b8a3388f5136"
format_stamp: "Formatted at 2026-06-19 18:50:10 on dist-test-slave-m9wg"
server_key: "4512017085bc64da101ab267e0829c0a"
server_key_iv: "f9aa9a356b97e4913808d79ae42c36f9"
server_key_version: "encryptionkey@0"
I20260619 18:50:10.455315 17989 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 18:50:10.458396 18004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:50:10.459614 17989 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 18:50:10.459753 17989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-0/wal
uuid: "8d3f261588f14e158cc4b8a3388f5136"
format_stamp: "Formatted at 2026-06-19 18:50:10 on dist-test-slave-m9wg"
server_key: "4512017085bc64da101ab267e0829c0a"
server_key_iv: "f9aa9a356b97e4913808d79ae42c36f9"
server_key_version: "encryptionkey@0"
I20260619 18:50:10.459910 17989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:50:10.482916 17989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:50:10.486577 17989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:50:10.486856 17989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:50:10.487772 17989 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:50:10.489029 17989 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:50:10.489099 17989 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:50:10.489184 17989 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:50:10.489224 17989 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:50:10.518352 17989 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.65:46855
I20260619 18:50:10.518433 18116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.65:46855 every 8 connection(s)
I20260619 18:50:10.519585 17989 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-0/data/info.pb
I20260619 18:50:10.529794 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 17989
I20260619 18:50:10.530025 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-0/wal/instance
I20260619 18:50:10.530815 4045 external_mini_cluster.cc:1428] Setting key 6f382b5aaf964ef03a30984dcaa8b620
Jun 19 18:50:10 dist-test-slave-m9wg krb5kdc[17883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895010, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.65@KRBTEST.COM for kudu/127.3.243.126@KRBTEST.COM
I20260619 18:50:10.538146 18119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:10.521876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.65:46865 (local address 127.3.243.126:39697)
0619 18:50:10.522259 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0619 18:50:10.522264 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:50:10.523241 (+ 977us) server_negotiation.cc:407] Connection header received
0619 18:50:10.524996 (+ 1755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:50:10.525000 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:50:10.525074 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:50:10.525156 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:50:10.526487 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:10.527168 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:10.528622 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:10.529004 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:10.529090 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:10.532823 (+ 3733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:50:10.532828 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:50:10.532831 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:50:10.532859 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:50:10.535161 (+ 2302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:10.535933 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:10.535938 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:10.535941 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:10.536094 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:10.536570 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:10.536576 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:10.536579 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:10.536869 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:50:10.537034 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0619 18:50:10.537741 (+ 707us) server_negotiation.cc:299] Negotiation successful
0619 18:50:10.537901 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":176,"threads_started":1}
I20260619 18:50:10.539400 18120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:10.522439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:39697 (local address 127.3.243.65:46865)
0619 18:50:10.523086 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:10.523123 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 18:50:10.524740 (+ 1617us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:50:10.525339 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:50:10.525347 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:50:10.525833 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:50:10.526361 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:50:10.526376 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:50:10.527272 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:50:10.527276 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:50:10.528504 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:50:10.528525 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:50:10.529157 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:50:10.529161 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:50:10.529340 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:10.530013 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:50:10.530043 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:50:10.532623 (+ 2580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:50:10.535287 (+ 2664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:50:10.535295 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:50:10.535310 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:50:10.535823 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:50:10.536206 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:50:10.536210 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:50:10.536213 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:50:10.536459 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:50:10.536999 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:50:10.537016 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:50:10.537385 (+ 369us) client_negotiation.cc:769] Sending connection context
0619 18:50:10.537656 (+ 271us) client_negotiation.cc:240] Negotiation successful
0619 18:50:10.538127 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":231,"threads_started":1}
I20260619 18:50:10.540758 18117 heartbeater.cc:347] Connected to a master server at 127.3.243.126:39697
I20260619 18:50:10.541025 18117 heartbeater.cc:464] Registering TS with master...
I20260619 18:50:10.541679 18117 heartbeater.cc:511] Master 127.3.243.126:39697 requested a full tablet report, sending...
I20260619 18:50:10.543540 17928 ts_manager.cc:194] Registered new tserver with Master: 8d3f261588f14e158cc4b8a3388f5136 (127.3.243.65:46855)
I20260619 18:50:10.544916 17928 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.65@KRBTEST.COM'} at 127.3.243.65:46865
WARNING: no policy specified for kudu/127.3.243.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.66.keytab.
Entry for principal kudu/127.3.243.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.66.keytab.
I20260619 18:50:10.575258 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.66:0
--local_ip_for_outbound_sockets=127.3.243.66
--webserver_interface=127.3.243.66
--webserver_port=0
--tserver_master_addrs=127.3.243.126:39697
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.66.keytab
--principal=kudu/127.3.243.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:46639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:50:10.876745 18123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:50:10.877024 18123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:50:10.877151 18123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:50:10.883984 18123 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:50:10.884086 18123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:50:10.884253 18123 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.66
I20260619 18:50:10.893455 18123 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:46639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.66.keytab
--principal=kudu/127.3.243.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.243.66
--webserver_port=0
--tserver_master_addrs=127.3.243.126:39697
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.66
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:50:10.894587 18123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:50:10.895802 18123 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:50:10.907744 18128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:50:10.907768 18129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:50:10.907861 18131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:50:10.908771 18123 server_base.cc:1034] running on GCE node
I20260619 18:50:10.909314 18123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:50:10.910116 18123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:50:10.911317 18123 hybrid_clock.cc:648] HybridClock initialized: now 1781895010911260 us; error 49 us; skew 500 ppm
I20260619 18:50:10.911597 18123 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:50:10 dist-test-slave-m9wg krb5kdc[17883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895010, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:50:10.914469 18123 init.cc:377] Logged in from keytab as kudu/127.3.243.66@KRBTEST.COM (short username kudu)
I20260619 18:50:10.915720 18123 webserver.cc:460] Webserver started at http://127.3.243.66:45245/ using document root <none> and password file <none>
I20260619 18:50:10.916366 18123 fs_manager.cc:359] Metadata directory not provided
I20260619 18:50:10.916463 18123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:50:10.916699 18123 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:50:10.919361 18123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-1/data/instance:
uuid: "fa3486d01e9045dcba230304040c37f1"
format_stamp: "Formatted at 2026-06-19 18:50:10 on dist-test-slave-m9wg"
server_key: "6c3293924cc918b396555e43d5f1d38d"
server_key_iv: "9838ff3f14c6f992483f7f08be0ff4a7"
server_key_version: "encryptionkey@0"
I20260619 18:50:10.920032 18123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-1/wal/instance:
uuid: "fa3486d01e9045dcba230304040c37f1"
format_stamp: "Formatted at 2026-06-19 18:50:10 on dist-test-slave-m9wg"
server_key: "6c3293924cc918b396555e43d5f1d38d"
server_key_iv: "9838ff3f14c6f992483f7f08be0ff4a7"
server_key_version: "encryptionkey@0"
I20260619 18:50:10.924604 18123 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 18:50:10.927618 18138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:50:10.928848 18123 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 18:50:10.928994 18123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-1/wal
uuid: "fa3486d01e9045dcba230304040c37f1"
format_stamp: "Formatted at 2026-06-19 18:50:10 on dist-test-slave-m9wg"
server_key: "6c3293924cc918b396555e43d5f1d38d"
server_key_iv: "9838ff3f14c6f992483f7f08be0ff4a7"
server_key_version: "encryptionkey@0"
I20260619 18:50:10.929152 18123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:50:10.969945 18123 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:50:10.973666 18123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:50:10.973989 18123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:50:10.974957 18123 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:50:10.976202 18123 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:50:10.976282 18123 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:50:10.976395 18123 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:50:10.976447 18123 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:50:11.004868 18123 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.66:46229
I20260619 18:50:11.004916 18250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.66:46229 every 8 connection(s)
I20260619 18:50:11.006026 18123 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-1/data/info.pb
I20260619 18:50:11.013484 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 18123
I20260619 18:50:11.013636 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-1/wal/instance
I20260619 18:50:11.014051 4045 external_mini_cluster.cc:1428] Setting key 4618b9b866e33299bc7f7469ffdbf9a7
Jun 19 18:50:11 dist-test-slave-m9wg krb5kdc[17883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895010, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.66@KRBTEST.COM for kudu/127.3.243.126@KRBTEST.COM
I20260619 18:50:11.024730 18119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:11.008317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.66:49509 (local address 127.3.243.126:39697)
0619 18:50:11.008497 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 18:50:11.008502 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:50:11.009531 (+ 1029us) server_negotiation.cc:407] Connection header received
0619 18:50:11.011245 (+ 1714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:50:11.011251 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:50:11.011332 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:50:11.011425 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:50:11.012860 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:11.013582 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:11.015196 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:11.015693 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:11.015818 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:11.019497 (+ 3679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:50:11.019501 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:50:11.019504 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:50:11.019533 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:50:11.021931 (+ 2398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:11.022587 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:11.022591 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:11.022593 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:11.022719 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:11.023114 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:11.023118 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:11.023120 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:11.023365 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:50:11.023514 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 18:50:11.024133 (+ 619us) server_negotiation.cc:299] Negotiation successful
0619 18:50:11.024385 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":53}
I20260619 18:50:11.025527 18253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:11.008739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:39697 (local address 127.3.243.66:49509)
0619 18:50:11.009365 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:11.009412 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 18:50:11.010986 (+ 1574us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:50:11.011648 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:50:11.011656 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:50:11.012137 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:50:11.012694 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:50:11.012711 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:50:11.013713 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:50:11.013719 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:50:11.015064 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:50:11.015077 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:50:11.015902 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:50:11.015910 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:50:11.016213 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:11.016829 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:50:11.016853 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:50:11.019308 (+ 2455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:50:11.022057 (+ 2749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:50:11.022066 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:50:11.022081 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:50:11.022472 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:50:11.022818 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:50:11.022822 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:50:11.022826 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:50:11.023003 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:50:11.023488 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:50:11.023495 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:50:11.023783 (+ 288us) client_negotiation.cc:769] Sending connection context
0619 18:50:11.023998 (+ 215us) client_negotiation.cc:240] Negotiation successful
0619 18:50:11.024474 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":234,"threads_started":1}
I20260619 18:50:11.026795 18251 heartbeater.cc:347] Connected to a master server at 127.3.243.126:39697
I20260619 18:50:11.027093 18251 heartbeater.cc:464] Registering TS with master...
I20260619 18:50:11.027860 18251 heartbeater.cc:511] Master 127.3.243.126:39697 requested a full tablet report, sending...
I20260619 18:50:11.029323 17928 ts_manager.cc:194] Registered new tserver with Master: fa3486d01e9045dcba230304040c37f1 (127.3.243.66:46229)
I20260619 18:50:11.030320 17928 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.66@KRBTEST.COM'} at 127.3.243.66:49509
WARNING: no policy specified for kudu/127.3.243.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.67.keytab.
Entry for principal kudu/127.3.243.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.67.keytab.
I20260619 18:50:11.059250 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.67:0
--local_ip_for_outbound_sockets=127.3.243.67
--webserver_interface=127.3.243.67
--webserver_port=0
--tserver_master_addrs=127.3.243.126:39697
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.67.keytab
--principal=kudu/127.3.243.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:46639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:50:11.355317 18256 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:50:11.355566 18256 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:50:11.355683 18256 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:50:11.362676 18256 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:50:11.362773 18256 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:50:11.362933 18256 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.67
I20260619 18:50:11.372298 18256 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:46639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.67.keytab
--principal=kudu/127.3.243.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.243.67
--webserver_port=0
--tserver_master_addrs=127.3.243.126:39697
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.67
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:50:11.373490 18256 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:50:11.374706 18256 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:50:11.386806 18261 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:50:11.386921 18262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:50:11.387081 18264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:50:11.387817 18256 server_base.cc:1034] running on GCE node
I20260619 18:50:11.388399 18256 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:50:11.389215 18256 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:50:11.390410 18256 hybrid_clock.cc:648] HybridClock initialized: now 1781895011390351 us; error 53 us; skew 500 ppm
I20260619 18:50:11.390686 18256 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:50:11 dist-test-slave-m9wg krb5kdc[17883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895011, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:50:11.393494 18256 init.cc:377] Logged in from keytab as kudu/127.3.243.67@KRBTEST.COM (short username kudu)
I20260619 18:50:11.394799 18256 webserver.cc:460] Webserver started at http://127.3.243.67:38533/ using document root <none> and password file <none>
I20260619 18:50:11.395488 18256 fs_manager.cc:359] Metadata directory not provided
I20260619 18:50:11.395601 18256 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:50:11.395857 18256 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:50:11.398743 18256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-2/data/instance:
uuid: "a87fb8ce1b304e8e9dbc3e969e690d9b"
format_stamp: "Formatted at 2026-06-19 18:50:11 on dist-test-slave-m9wg"
server_key: "38f5df90f25296131d3953cc271cb82b"
server_key_iv: "e66a5563157d00381fb091c0243bea8d"
server_key_version: "encryptionkey@0"
I20260619 18:50:11.399474 18256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-2/wal/instance:
uuid: "a87fb8ce1b304e8e9dbc3e969e690d9b"
format_stamp: "Formatted at 2026-06-19 18:50:11 on dist-test-slave-m9wg"
server_key: "38f5df90f25296131d3953cc271cb82b"
server_key_iv: "e66a5563157d00381fb091c0243bea8d"
server_key_version: "encryptionkey@0"
I20260619 18:50:11.404292 18256 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260619 18:50:11.407491 18271 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:50:11.408766 18256 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:50:11.408906 18256 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-2/wal
uuid: "a87fb8ce1b304e8e9dbc3e969e690d9b"
format_stamp: "Formatted at 2026-06-19 18:50:11 on dist-test-slave-m9wg"
server_key: "38f5df90f25296131d3953cc271cb82b"
server_key_iv: "e66a5563157d00381fb091c0243bea8d"
server_key_version: "encryptionkey@0"
I20260619 18:50:11.409075 18256 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:50:11.429549 18256 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:50:11.433089 18256 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:50:11.433322 18256 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:50:11.434192 18256 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:50:11.435413 18256 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:50:11.435487 18256 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:50:11.435593 18256 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:50:11.435642 18256 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:50:11.463944 18256 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.67:34767
I20260619 18:50:11.464021 18383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.67:34767 every 8 connection(s)
I20260619 18:50:11.465103 18256 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-2/data/info.pb
I20260619 18:50:11.466598 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 18256
I20260619 18:50:11.466760 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-2/wal/instance
I20260619 18:50:11.467187 4045 external_mini_cluster.cc:1428] Setting key 12dff5bad878bc39371379e60d369201
Jun 19 18:50:11 dist-test-slave-m9wg krb5kdc[17883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895011, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.67@KRBTEST.COM for kudu/127.3.243.126@KRBTEST.COM
I20260619 18:50:11.483410 18119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:11.467427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.67:60957 (local address 127.3.243.126:39697)
0619 18:50:11.467646 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0619 18:50:11.467653 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:50:11.468912 (+ 1259us) server_negotiation.cc:407] Connection header received
0619 18:50:11.471020 (+ 2108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:50:11.471025 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:50:11.471109 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:50:11.471189 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:50:11.472646 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:11.473344 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:11.474770 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:11.475172 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:11.475274 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:11.478291 (+ 3017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:50:11.478295 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:50:11.478298 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:50:11.478329 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:50:11.480547 (+ 2218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:11.481299 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:11.481303 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:11.481306 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:11.481422 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:11.481822 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:11.481825 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:11.481828 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:11.482104 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:50:11.482235 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 18:50:11.482860 (+ 625us) server_negotiation.cc:299] Negotiation successful
0619 18:50:11.483085 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 18:50:11.484269 18386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:11.467983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:39697 (local address 127.3.243.67:60957)
0619 18:50:11.468770 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:11.468812 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 18:50:11.470784 (+ 1972us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:50:11.471419 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:50:11.471426 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:50:11.471902 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:50:11.472492 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:50:11.472511 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:50:11.473482 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:50:11.473486 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:50:11.474643 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:50:11.474654 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:50:11.475368 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:50:11.475375 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:50:11.475585 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:11.476102 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:50:11.476123 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:50:11.478124 (+ 2001us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:50:11.480664 (+ 2540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:50:11.480672 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:50:11.480686 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:50:11.481186 (+ 500us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:50:11.481518 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:50:11.481523 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:50:11.481527 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:50:11.481708 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:50:11.482221 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:50:11.482228 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:50:11.482511 (+ 283us) client_negotiation.cc:769] Sending connection context
0619 18:50:11.482732 (+ 221us) client_negotiation.cc:240] Negotiation successful
0619 18:50:11.483176 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":318,"threads_started":1}
I20260619 18:50:11.485385 18384 heartbeater.cc:347] Connected to a master server at 127.3.243.126:39697
I20260619 18:50:11.485641 18384 heartbeater.cc:464] Registering TS with master...
I20260619 18:50:11.486250 18384 heartbeater.cc:511] Master 127.3.243.126:39697 requested a full tablet report, sending...
I20260619 18:50:11.487471 17928 ts_manager.cc:194] Registered new tserver with Master: a87fb8ce1b304e8e9dbc3e969e690d9b (127.3.243.67:34767)
I20260619 18:50:11.488262 17928 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.67@KRBTEST.COM'} at 127.3.243.67:60957
I20260619 18:50:11.494133 4045 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 18:50:11.509378 18119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:11.498289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54854 (local address 127.3.243.126:39697)
0619 18:50:11.498424 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 18:50:11.498428 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:50:11.498862 (+ 434us) server_negotiation.cc:407] Connection header received
0619 18:50:11.498982 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:50:11.498985 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:50:11.499041 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:50:11.499112 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:50:11.500411 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:11.501124 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:11.502357 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:11.502737 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:11.502839 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:11.505015 (+ 2176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:50:11.505019 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:50:11.505022 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:50:11.505049 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:50:11.507339 (+ 2290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:11.507898 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:11.507902 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:11.507904 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:11.508040 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:11.508340 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:11.508342 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:11.508344 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:11.508618 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:50:11.508783 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0619 18:50:11.509015 (+ 232us) server_negotiation.cc:299] Negotiation successful
0619 18:50:11.509157 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 18:50:11.512753 17928 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54854:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:50:11.519945 17928 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:50:11.547466 18117 heartbeater.cc:503] Master 127.3.243.126:39697 was elected leader, sending a full tablet report...
I20260619 18:50:11.553732 18399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:11.532566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36816 (local address 127.3.243.65:46855)
0619 18:50:11.543974 (+ 11408us) server_negotiation.cc:206] Beginning negotiation
0619 18:50:11.543984 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:50:11.545333 (+ 1349us) server_negotiation.cc:407] Connection header received
0619 18:50:11.545436 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:50:11.545444 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:50:11.545697 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:50:11.546137 (+ 440us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:50:11.547362 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:11.548769 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:11.551149 (+ 2380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:11.552499 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:11.552627 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:11.552812 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0619 18:50:11.553267 (+ 455us) server_negotiation.cc:299] Negotiation successful
0619 18:50:11.553423 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11235,"thread_start_us":11095,"threads_started":1}
I20260619 18:50:11.554049 18397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:11.532085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.65:46855 (local address 127.0.0.1:36816)
0619 18:50:11.544908 (+ 12823us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:11.544953 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 18:50:11.545130 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:50:11.545951 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:50:11.545958 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:50:11.545993 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:50:11.547145 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:50:11.547158 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:50:11.548940 (+ 1782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:50:11.548945 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:50:11.550936 (+ 1991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:50:11.550948 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:50:11.552925 (+ 1977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:50:11.552929 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:50:11.553061 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:11.553097 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 18:50:11.553202 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 18:50:11.553387 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12662,"mutex_wait_us":409,"thread_start_us":12522,"threads_started":1}
I20260619 18:50:11.554339 18398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:11.532124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33404 (local address 127.3.243.66:46229)
0619 18:50:11.544304 (+ 12180us) server_negotiation.cc:206] Beginning negotiation
0619 18:50:11.544313 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:50:11.545055 (+ 742us) server_negotiation.cc:407] Connection header received
0619 18:50:11.545462 (+ 407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:50:11.545470 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:50:11.545719 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:50:11.545876 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:50:11.546669 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:11.548847 (+ 2178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:11.551618 (+ 2771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:11.553035 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:11.553140 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:11.553312 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0619 18:50:11.553856 (+ 544us) server_negotiation.cc:299] Negotiation successful
0619 18:50:11.554032 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11994,"thread_start_us":11842,"threads_started":1}
I20260619 18:50:11.554939 18396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:11.531956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.66:46229 (local address 127.0.0.1:33404)
0619 18:50:11.544910 (+ 12954us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:11.544952 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 18:50:11.545163 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:50:11.545992 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:50:11.545997 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:50:11.546032 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:50:11.546470 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:50:11.546483 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:50:11.549107 (+ 2624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:50:11.549112 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:50:11.551431 (+ 2319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:50:11.551447 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:50:11.553495 (+ 2048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:50:11.553501 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:50:11.553662 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:11.553697 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 18:50:11.554464 (+ 767us) client_negotiation.cc:240] Negotiation successful
0619 18:50:11.554585 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12692,"mutex_wait_us":98,"thread_start_us":12481,"threads_started":1}
I20260619 18:50:11.555788 18401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:11.546383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.67:34767 (local address 127.0.0.1:33650)
0619 18:50:11.547777 (+ 1394us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:11.547799 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 18:50:11.547900 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:50:11.549278 (+ 1378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:50:11.549289 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:50:11.549318 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:50:11.549746 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:50:11.549758 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:50:11.551613 (+ 1855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:50:11.551618 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:50:11.553609 (+ 1991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:50:11.553620 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:50:11.555117 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:50:11.555124 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:50:11.555291 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:11.555323 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 18:50:11.555393 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 18:50:11.555505 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"thread_start_us":396,"threads_started":1}
I20260619 18:50:11.556949 18400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:11.545607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33650 (local address 127.3.243.67:34767)
0619 18:50:11.548411 (+ 2804us) server_negotiation.cc:206] Beginning negotiation
0619 18:50:11.548419 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:50:11.548449 (+ 30us) server_negotiation.cc:407] Connection header received
0619 18:50:11.548546 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:50:11.548553 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:50:11.548760 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:50:11.548933 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:50:11.549942 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:11.551457 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:11.553774 (+ 2317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:11.554911 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:11.556202 (+ 1291us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:11.556425 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0619 18:50:11.556501 (+ 76us) server_negotiation.cc:299] Negotiation successful
0619 18:50:11.556643 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1589,"thread_start_us":1442,"threads_started":1}
I20260619 18:50:11.558102 18052 tablet_service.cc:1467] Processing CreateTablet for tablet c5e56d3f59e1427aa400dbc8ac1c0543 (DEFAULT_TABLE table=test-table [id=cfe987178d114473b0070ec719853886]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:50:11.558358 18186 tablet_service.cc:1467] Processing CreateTablet for tablet c5e56d3f59e1427aa400dbc8ac1c0543 (DEFAULT_TABLE table=test-table [id=cfe987178d114473b0070ec719853886]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:50:11.559237 18052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5e56d3f59e1427aa400dbc8ac1c0543. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:50:11.559685 18186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5e56d3f59e1427aa400dbc8ac1c0543. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:50:11.560268 18319 tablet_service.cc:1467] Processing CreateTablet for tablet c5e56d3f59e1427aa400dbc8ac1c0543 (DEFAULT_TABLE table=test-table [id=cfe987178d114473b0070ec719853886]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:50:11.561522 18319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5e56d3f59e1427aa400dbc8ac1c0543. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:50:11.576372 18403 tablet_bootstrap.cc:492] T c5e56d3f59e1427aa400dbc8ac1c0543 P fa3486d01e9045dcba230304040c37f1: Bootstrap starting.
I20260619 18:50:11.576960 18404 tablet_bootstrap.cc:492] T c5e56d3f59e1427aa400dbc8ac1c0543 P a87fb8ce1b304e8e9dbc3e969e690d9b: Bootstrap starting.
I20260619 18:50:11.579384 18403 tablet_bootstrap.cc:654] T c5e56d3f59e1427aa400dbc8ac1c0543 P fa3486d01e9045dcba230304040c37f1: Neither blocks nor log segments found. Creating new log.
I20260619 18:50:11.580778 18404 tablet_bootstrap.cc:654] T c5e56d3f59e1427aa400dbc8ac1c0543 P a87fb8ce1b304e8e9dbc3e969e690d9b: Neither blocks nor log segments found. Creating new log.
I20260619 18:50:11.581411 18402 tablet_bootstrap.cc:492] T c5e56d3f59e1427aa400dbc8ac1c0543 P 8d3f261588f14e158cc4b8a3388f5136: Bootstrap starting.
I20260619 18:50:11.583812 18402 tablet_bootstrap.cc:654] T c5e56d3f59e1427aa400dbc8ac1c0543 P 8d3f261588f14e158cc4b8a3388f5136: Neither blocks nor log segments found. Creating new log.
I20260619 18:50:11.584475 18403 log.cc:824] T c5e56d3f59e1427aa400dbc8ac1c0543 P fa3486d01e9045dcba230304040c37f1: Log is configured to *not* fsync() on all Append() calls
I20260619 18:50:11.584990 18402 log.cc:824] T c5e56d3f59e1427aa400dbc8ac1c0543 P 8d3f261588f14e158cc4b8a3388f5136: Log is configured to *not* fsync() on all Append() calls
I20260619 18:50:11.593250 18404 log.cc:824] T c5e56d3f59e1427aa400dbc8ac1c0543 P a87fb8ce1b304e8e9dbc3e969e690d9b: Log is configured to *not* fsync() on all Append() calls
I20260619 18:50:11.595247 18403 tablet_bootstrap.cc:492] T c5e56d3f59e1427aa400dbc8ac1c0543 P fa3486d01e9045dcba230304040c37f1: No bootstrap required, opened a new log
I20260619 18:50:11.595546 18403 ts_tablet_manager.cc:1397] T c5e56d3f59e1427aa400dbc8ac1c0543 P fa3486d01e9045dcba230304040c37f1: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.007s
I20260619 18:50:11.596756 18404 tablet_bootstrap.cc:492] T c5e56d3f59e1427aa400dbc8ac1c0543 P a87fb8ce1b304e8e9dbc3e969e690d9b: No bootstrap required, opened a new log
I20260619 18:50:11.597012 18404 ts_tablet_manager.cc:1397] T c5e56d3f59e1427aa400dbc8ac1c0543 P a87fb8ce1b304e8e9dbc3e969e690d9b: Time spent bootstrapping tablet: real 0.020s user 0.007s sys 0.000s
I20260619 18:50:11.597218 18402 tablet_bootstrap.cc:492] T c5e56d3f59e1427aa400dbc8ac1c0543 P 8d3f261588f14e158cc4b8a3388f5136: No bootstrap required, opened a new log
I20260619 18:50:11.597476 18402 ts_tablet_manager.cc:1397] T c5e56d3f59e1427aa400dbc8ac1c0543 P 8d3f261588f14e158cc4b8a3388f5136: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.008s
I20260619 18:50:11.606801 18403 raft_consensus.cc:348] T c5e56d3f59e1427aa400dbc8ac1c0543 P fa3486d01e9045dcba230304040c37f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d3f261588f14e158cc4b8a3388f5136" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 46855 } } peers { permanent_uuid: "fa3486d01e9045dcba230304040c37f1" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 46229 } } peers { permanent_uuid: "a87fb8ce1b304e8e9dbc3e969e690d9b" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 34767 } }
I20260619 18:50:11.607219 18403 raft_consensus.cc:374] T c5e56d3f59e1427aa400dbc8ac1c0543 P fa3486d01e9045dcba230304040c37f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:50:11.607302 18403 raft_consensus.cc:729] T c5e56d3f59e1427aa400dbc8ac1c0543 P fa3486d01e9045dcba230304040c37f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa3486d01e9045dcba230304040c37f1, State: Initialized, Role: FOLLOWER
I20260619 18:50:11.607329 18402 raft_consensus.cc:348] T c5e56d3f59e1427aa400dbc8ac1c0543 P 8d3f261588f14e158cc4b8a3388f5136 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d3f261588f14e158cc4b8a3388f5136" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 46855 } } peers { permanent_uuid: "fa3486d01e9045dcba230304040c37f1" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 46229 } } peers { permanent_uuid: "a87fb8ce1b304e8e9dbc3e969e690d9b" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 34767 } }
I20260619 18:50:11.607709 18402 raft_consensus.cc:374] T c5e56d3f59e1427aa400dbc8ac1c0543 P 8d3f261588f14e158cc4b8a3388f5136 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:50:11.607329 18404 raft_consensus.cc:348] T c5e56d3f59e1427aa400dbc8ac1c0543 P a87fb8ce1b304e8e9dbc3e969e690d9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d3f261588f14e158cc4b8a3388f5136" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 46855 } } peers { permanent_uuid: "fa3486d01e9045dcba230304040c37f1" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 46229 } } peers { permanent_uuid: "a87fb8ce1b304e8e9dbc3e969e690d9b" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 34767 } }
I20260619 18:50:11.607801 18402 raft_consensus.cc:729] T c5e56d3f59e1427aa400dbc8ac1c0543 P 8d3f261588f14e158cc4b8a3388f5136 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d3f261588f14e158cc4b8a3388f5136, State: Initialized, Role: FOLLOWER
I20260619 18:50:11.607843 18404 raft_consensus.cc:374] T c5e56d3f59e1427aa400dbc8ac1c0543 P a87fb8ce1b304e8e9dbc3e969e690d9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:50:11.607918 18404 raft_consensus.cc:729] T c5e56d3f59e1427aa400dbc8ac1c0543 P a87fb8ce1b304e8e9dbc3e969e690d9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a87fb8ce1b304e8e9dbc3e969e690d9b, State: Initialized, Role: FOLLOWER
I20260619 18:50:11.607893 18403 consensus_queue.cc:260] T c5e56d3f59e1427aa400dbc8ac1c0543 P fa3486d01e9045dcba230304040c37f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d3f261588f14e158cc4b8a3388f5136" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 46855 } } peers { permanent_uuid: "fa3486d01e9045dcba230304040c37f1" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 46229 } } peers { permanent_uuid: "a87fb8ce1b304e8e9dbc3e969e690d9b" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 34767 } }
I20260619 18:50:11.608395 18402 consensus_queue.cc:260] T c5e56d3f59e1427aa400dbc8ac1c0543 P 8d3f261588f14e158cc4b8a3388f5136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d3f261588f14e158cc4b8a3388f5136" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 46855 } } peers { permanent_uuid: "fa3486d01e9045dcba230304040c37f1" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 46229 } } peers { permanent_uuid: "a87fb8ce1b304e8e9dbc3e969e690d9b" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 34767 } }
I20260619 18:50:11.608525 18404 consensus_queue.cc:260] T c5e56d3f59e1427aa400dbc8ac1c0543 P a87fb8ce1b304e8e9dbc3e969e690d9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d3f261588f14e158cc4b8a3388f5136" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 46855 } } peers { permanent_uuid: "fa3486d01e9045dcba230304040c37f1" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 46229 } } peers { permanent_uuid: "a87fb8ce1b304e8e9dbc3e969e690d9b" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 34767 } }
I20260619 18:50:11.611917 18404 ts_tablet_manager.cc:1428] T c5e56d3f59e1427aa400dbc8ac1c0543 P a87fb8ce1b304e8e9dbc3e969e690d9b: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260619 18:50:11.612376 18384 heartbeater.cc:503] Master 127.3.243.126:39697 was elected leader, sending a full tablet report...
I20260619 18:50:11.619524 18251 heartbeater.cc:503] Master 127.3.243.126:39697 was elected leader, sending a full tablet report...
I20260619 18:50:11.620141 18403 ts_tablet_manager.cc:1428] T c5e56d3f59e1427aa400dbc8ac1c0543 P fa3486d01e9045dcba230304040c37f1: Time spent starting tablet: real 0.024s user 0.012s sys 0.009s
I20260619 18:50:11.620376 18402 ts_tablet_manager.cc:1428] T c5e56d3f59e1427aa400dbc8ac1c0543 P 8d3f261588f14e158cc4b8a3388f5136: Time spent starting tablet: real 0.023s user 0.019s sys 0.004s
W20260619 18:50:11.716445 18385 tablet.cc:2369] T c5e56d3f59e1427aa400dbc8ac1c0543 P a87fb8ce1b304e8e9dbc3e969e690d9b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:50:11.736218 18409 raft_consensus.cc:482] T c5e56d3f59e1427aa400dbc8ac1c0543 P 8d3f261588f14e158cc4b8a3388f5136 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:50:11.736508 18409 raft_consensus.cc:504] T c5e56d3f59e1427aa400dbc8ac1c0543 P 8d3f261588f14e158cc4b8a3388f5136 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d3f261588f14e158cc4b8a3388f5136" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 46855 } } peers { permanent_uuid: "fa3486d01e9045dcba230304040c37f1" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 46229 } } peers { permanent_uuid: "a87fb8ce1b304e8e9dbc3e969e690d9b" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 34767 } }
I20260619 18:50:11.738319 18409 leader_election.cc:288] T c5e56d3f59e1427aa400dbc8ac1c0543 P 8d3f261588f14e158cc4b8a3388f5136 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa3486d01e9045dcba230304040c37f1 (127.3.243.66:46229), a87fb8ce1b304e8e9dbc3e969e690d9b (127.3.243.67:34767)
I20260619 18:50:11.745841 18398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:11.738744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.65:42875 (local address 127.3.243.66:46229)
0619 18:50:11.738916 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 18:50:11.738921 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:50:11.739454 (+ 533us) server_negotiation.cc:407] Connection header received
0619 18:50:11.739658 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:50:11.739663 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:50:11.739741 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:50:11.739827 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:50:11.740468 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:11.741490 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:11.743941 (+ 2451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:11.744966 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:11.745054 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:11.745103 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0619 18:50:11.745542 (+ 439us) server_negotiation.cc:299] Negotiation successful
0619 18:50:11.745643 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 18:50:11.746023 18411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:11.738710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.66:46229 (local address 127.3.243.65:42875)
0619 18:50:11.739353 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:11.739372 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 18:50:11.739499 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:50:11.739935 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:50:11.739940 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:50:11.739967 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:50:11.740330 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:50:11.740344 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:50:11.741637 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:50:11.741643 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:50:11.743757 (+ 2114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:50:11.743772 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:50:11.745240 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:50:11.745252 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:50:11.745366 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:11.745394 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 18:50:11.745487 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 18:50:11.745613 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"mutex_wait_us":21,"thread_start_us":346,"threads_started":1}
I20260619 18:50:11.746172 18400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:11.739619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.65:40141 (local address 127.3.243.67:34767)
0619 18:50:11.739837 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0619 18:50:11.739842 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:50:11.740219 (+ 377us) server_negotiation.cc:407] Connection header received
0619 18:50:11.740392 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:50:11.740397 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:50:11.740474 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:50:11.740543 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:50:11.741141 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:11.742182 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:11.744599 (+ 2417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:11.745329 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:11.745442 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:11.745514 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 18:50:11.745837 (+ 323us) server_negotiation.cc:299] Negotiation successful
0619 18:50:11.745965 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 18:50:11.746296 18412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:11.739500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.67:34767 (local address 127.3.243.65:40141)
0619 18:50:11.740119 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:11.740132 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 18:50:11.740266 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:50:11.740582 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:50:11.740587 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:50:11.740611 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:50:11.740924 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:50:11.740935 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:50:11.742322 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:50:11.742328 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:50:11.744445 (+ 2117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:50:11.744459 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:50:11.745530 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:50:11.745536 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:50:11.745683 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:11.745715 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 18:50:11.745802 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 18:50:11.745913 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"mutex_wait_us":47,"thread_start_us":301,"threads_started":1}
I20260619 18:50:11.746937 18206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5e56d3f59e1427aa400dbc8ac1c0543" candidate_uuid: "8d3f261588f14e158cc4b8a3388f5136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa3486d01e9045dcba230304040c37f1" is_pre_election: true
I20260619 18:50:11.747283 18339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5e56d3f59e1427aa400dbc8ac1c0543" candidate_uuid: "8d3f261588f14e158cc4b8a3388f5136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a87fb8ce1b304e8e9dbc3e969e690d9b" is_pre_election: true
I20260619 18:50:11.747380 18206 raft_consensus.cc:2454] T c5e56d3f59e1427aa400dbc8ac1c0543 P fa3486d01e9045dcba230304040c37f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d3f261588f14e158cc4b8a3388f5136 in term 0.
I20260619 18:50:11.747680 18339 raft_consensus.cc:2454] T c5e56d3f59e1427aa400dbc8ac1c0543 P a87fb8ce1b304e8e9dbc3e969e690d9b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d3f261588f14e158cc4b8a3388f5136 in term 0.
I20260619 18:50:11.747974 18006 leader_election.cc:302] T c5e56d3f59e1427aa400dbc8ac1c0543 P 8d3f261588f14e158cc4b8a3388f5136 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d3f261588f14e158cc4b8a3388f5136, fa3486d01e9045dcba230304040c37f1; no voters:
I20260619 18:50:11.748323 18409 raft_consensus.cc:2781] T c5e56d3f59e1427aa400dbc8ac1c0543 P 8d3f261588f14e158cc4b8a3388f5136 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:50:11.748459 18409 raft_consensus.cc:482] T c5e56d3f59e1427aa400dbc8ac1c0543 P 8d3f261588f14e158cc4b8a3388f5136 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:50:11.748557 18409 raft_consensus.cc:3037] T c5e56d3f59e1427aa400dbc8ac1c0543 P 8d3f261588f14e158cc4b8a3388f5136 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:50:11.750610 18409 raft_consensus.cc:504] T c5e56d3f59e1427aa400dbc8ac1c0543 P 8d3f261588f14e158cc4b8a3388f5136 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d3f261588f14e158cc4b8a3388f5136" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 46855 } } peers { permanent_uuid: "fa3486d01e9045dcba230304040c37f1" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 46229 } } peers { permanent_uuid: "a87fb8ce1b304e8e9dbc3e969e690d9b" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 34767 } }
I20260619 18:50:11.751123 18409 leader_election.cc:288] T c5e56d3f59e1427aa400dbc8ac1c0543 P 8d3f261588f14e158cc4b8a3388f5136 [CANDIDATE]: Term 1 election: Requested vote from peers fa3486d01e9045dcba230304040c37f1 (127.3.243.66:46229), a87fb8ce1b304e8e9dbc3e969e690d9b (127.3.243.67:34767)
I20260619 18:50:11.751509 18206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5e56d3f59e1427aa400dbc8ac1c0543" candidate_uuid: "8d3f261588f14e158cc4b8a3388f5136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa3486d01e9045dcba230304040c37f1"
I20260619 18:50:11.751591 18339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5e56d3f59e1427aa400dbc8ac1c0543" candidate_uuid: "8d3f261588f14e158cc4b8a3388f5136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a87fb8ce1b304e8e9dbc3e969e690d9b"
I20260619 18:50:11.751673 18206 raft_consensus.cc:3037] T c5e56d3f59e1427aa400dbc8ac1c0543 P fa3486d01e9045dcba230304040c37f1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:50:11.751765 18339 raft_consensus.cc:3037] T c5e56d3f59e1427aa400dbc8ac1c0543 P a87fb8ce1b304e8e9dbc3e969e690d9b [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:50:11.753794 18339 raft_consensus.cc:2454] T c5e56d3f59e1427aa400dbc8ac1c0543 P a87fb8ce1b304e8e9dbc3e969e690d9b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d3f261588f14e158cc4b8a3388f5136 in term 1.
I20260619 18:50:11.753961 18206 raft_consensus.cc:2454] T c5e56d3f59e1427aa400dbc8ac1c0543 P fa3486d01e9045dcba230304040c37f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d3f261588f14e158cc4b8a3388f5136 in term 1.
I20260619 18:50:11.754123 18006 leader_election.cc:302] T c5e56d3f59e1427aa400dbc8ac1c0543 P 8d3f261588f14e158cc4b8a3388f5136 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d3f261588f14e158cc4b8a3388f5136, a87fb8ce1b304e8e9dbc3e969e690d9b; no voters:
I20260619 18:50:11.754379 18409 raft_consensus.cc:2781] T c5e56d3f59e1427aa400dbc8ac1c0543 P 8d3f261588f14e158cc4b8a3388f5136 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:50:11.754863 18409 raft_consensus.cc:686] T c5e56d3f59e1427aa400dbc8ac1c0543 P 8d3f261588f14e158cc4b8a3388f5136 [term 1 LEADER]: Becoming Leader. State: Replica: 8d3f261588f14e158cc4b8a3388f5136, State: Running, Role: LEADER
I20260619 18:50:11.755283 18409 consensus_queue.cc:237] T c5e56d3f59e1427aa400dbc8ac1c0543 P 8d3f261588f14e158cc4b8a3388f5136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d3f261588f14e158cc4b8a3388f5136" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 46855 } } peers { permanent_uuid: "fa3486d01e9045dcba230304040c37f1" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 46229 } } peers { permanent_uuid: "a87fb8ce1b304e8e9dbc3e969e690d9b" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 34767 } }
W20260619 18:50:11.757794 18252 tablet.cc:2369] T c5e56d3f59e1427aa400dbc8ac1c0543 P fa3486d01e9045dcba230304040c37f1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:50:11.759240 17928 catalog_manager.cc:5495] T c5e56d3f59e1427aa400dbc8ac1c0543 P 8d3f261588f14e158cc4b8a3388f5136 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d3f261588f14e158cc4b8a3388f5136 (127.3.243.65). New cstate: current_term: 1 leader_uuid: "8d3f261588f14e158cc4b8a3388f5136" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d3f261588f14e158cc4b8a3388f5136" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 46855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fa3486d01e9045dcba230304040c37f1" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 46229 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a87fb8ce1b304e8e9dbc3e969e690d9b" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 34767 } health_report { overall_health: UNKNOWN } } }
W20260619 18:50:11.771581 18118 tablet.cc:2369] T c5e56d3f59e1427aa400dbc8ac1c0543 P 8d3f261588f14e158cc4b8a3388f5136: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:50:11.793776 18399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:11.789218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36822 (local address 127.3.243.65:46855)
0619 18:50:11.789378 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 18:50:11.789382 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:50:11.789397 (+ 15us) server_negotiation.cc:407] Connection header received
0619 18:50:11.789568 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:50:11.789571 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:50:11.789648 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:50:11.789756 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:50:11.790041 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:11.790706 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:11.791973 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:11.792420 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:11.792508 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:11.792875 (+ 367us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:50:11.793345 (+ 470us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:50:11.793442 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 18:50:11.793529 (+ 87us) server_negotiation.cc:299] Negotiation successful
0619 18:50:11.793589 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 18:50:11.801010 18339 raft_consensus.cc:1261] T c5e56d3f59e1427aa400dbc8ac1c0543 P a87fb8ce1b304e8e9dbc3e969e690d9b [term 1 FOLLOWER]: Refusing update from remote peer 8d3f261588f14e158cc4b8a3388f5136: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:50:11.801083 18206 raft_consensus.cc:1261] T c5e56d3f59e1427aa400dbc8ac1c0543 P fa3486d01e9045dcba230304040c37f1 [term 1 FOLLOWER]: Refusing update from remote peer 8d3f261588f14e158cc4b8a3388f5136: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:50:11.801847 18413 consensus_queue.cc:1035] T c5e56d3f59e1427aa400dbc8ac1c0543 P 8d3f261588f14e158cc4b8a3388f5136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a87fb8ce1b304e8e9dbc3e969e690d9b" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 34767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:50:11.802171 18409 consensus_queue.cc:1035] T c5e56d3f59e1427aa400dbc8ac1c0543 P 8d3f261588f14e158cc4b8a3388f5136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa3486d01e9045dcba230304040c37f1" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 46229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:50:11.824327 17928 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54854:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:50:11.824556 17928 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54854:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:50:11.827807 17928 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 22ae91d05d6a4613906a3966068df987: Sending DeleteTablet for 3 replicas of tablet c5e56d3f59e1427aa400dbc8ac1c0543
I20260619 18:50:11.828845 18052 tablet_service.cc:1514] Processing DeleteTablet for tablet c5e56d3f59e1427aa400dbc8ac1c0543 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:50:11 UTC) from {username='kudu', principal='kudu/127.3.243.126@KRBTEST.COM'} at 127.0.0.1:36816
I20260619 18:50:11.828908 18319 tablet_service.cc:1514] Processing DeleteTablet for tablet c5e56d3f59e1427aa400dbc8ac1c0543 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:50:11 UTC) from {username='kudu', principal='kudu/127.3.243.126@KRBTEST.COM'} at 127.0.0.1:33650
I20260619 18:50:11.828964 18186 tablet_service.cc:1514] Processing DeleteTablet for tablet c5e56d3f59e1427aa400dbc8ac1c0543 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:50:11 UTC) from {username='kudu', principal='kudu/127.3.243.126@KRBTEST.COM'} at 127.0.0.1:33404
I20260619 18:50:11.829975 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 17989
I20260619 18:50:11.841225 18427 tablet_replica.cc:330] T c5e56d3f59e1427aa400dbc8ac1c0543 P a87fb8ce1b304e8e9dbc3e969e690d9b: stopping tablet replica
I20260619 18:50:11.841662 18427 raft_consensus.cc:2229] T c5e56d3f59e1427aa400dbc8ac1c0543 P a87fb8ce1b304e8e9dbc3e969e690d9b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:50:11.842187 18427 raft_consensus.cc:2258] T c5e56d3f59e1427aa400dbc8ac1c0543 P a87fb8ce1b304e8e9dbc3e969e690d9b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:50:11.844398 18427 ts_tablet_manager.cc:1905] T c5e56d3f59e1427aa400dbc8ac1c0543 P a87fb8ce1b304e8e9dbc3e969e690d9b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:50:11.851486 18427 ts_tablet_manager.cc:1918] T c5e56d3f59e1427aa400dbc8ac1c0543 P a87fb8ce1b304e8e9dbc3e969e690d9b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:50:11.851642 18427 log.cc:1196] T c5e56d3f59e1427aa400dbc8ac1c0543 P a87fb8ce1b304e8e9dbc3e969e690d9b: Deleting WAL directory at /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-2/wal/wals/c5e56d3f59e1427aa400dbc8ac1c0543
I20260619 18:50:11.852631 18428 tablet_replica.cc:330] T c5e56d3f59e1427aa400dbc8ac1c0543 P fa3486d01e9045dcba230304040c37f1: stopping tablet replica
I20260619 18:50:11.853031 18428 raft_consensus.cc:2229] T c5e56d3f59e1427aa400dbc8ac1c0543 P fa3486d01e9045dcba230304040c37f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:50:11.853506 18428 raft_consensus.cc:2258] T c5e56d3f59e1427aa400dbc8ac1c0543 P fa3486d01e9045dcba230304040c37f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:50:11.853595 18426 tablet_replica.cc:330] T c5e56d3f59e1427aa400dbc8ac1c0543 P 8d3f261588f14e158cc4b8a3388f5136: stopping tablet replica
I20260619 18:50:11.853965 18426 raft_consensus.cc:2229] T c5e56d3f59e1427aa400dbc8ac1c0543 P 8d3f261588f14e158cc4b8a3388f5136 [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:50:11.854591 18426 raft_consensus.cc:2258] T c5e56d3f59e1427aa400dbc8ac1c0543 P 8d3f261588f14e158cc4b8a3388f5136 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:50:11.852141 18427 ts_tablet_manager.cc:1939] T c5e56d3f59e1427aa400dbc8ac1c0543 P a87fb8ce1b304e8e9dbc3e969e690d9b: Deleting consensus metadata
I20260619 18:50:11.855477 18428 ts_tablet_manager.cc:1905] T c5e56d3f59e1427aa400dbc8ac1c0543 P fa3486d01e9045dcba230304040c37f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:50:11.856554 17913 catalog_manager.cc:4841] TS a87fb8ce1b304e8e9dbc3e969e690d9b (127.3.243.67:34767): tablet c5e56d3f59e1427aa400dbc8ac1c0543 (table test-table [id=cfe987178d114473b0070ec719853886]) successfully deleted
I20260619 18:50:11.857385 18426 ts_tablet_manager.cc:1905] T c5e56d3f59e1427aa400dbc8ac1c0543 P 8d3f261588f14e158cc4b8a3388f5136: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:50:11.861872 18428 ts_tablet_manager.cc:1918] T c5e56d3f59e1427aa400dbc8ac1c0543 P fa3486d01e9045dcba230304040c37f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:50:11.862020 18428 log.cc:1196] T c5e56d3f59e1427aa400dbc8ac1c0543 P fa3486d01e9045dcba230304040c37f1: Deleting WAL directory at /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-1/wal/wals/c5e56d3f59e1427aa400dbc8ac1c0543
I20260619 18:50:11.862469 18428 ts_tablet_manager.cc:1939] T c5e56d3f59e1427aa400dbc8ac1c0543 P fa3486d01e9045dcba230304040c37f1: Deleting consensus metadata
I20260619 18:50:11.864070 17913 catalog_manager.cc:4841] TS fa3486d01e9045dcba230304040c37f1 (127.3.243.66:46229): tablet c5e56d3f59e1427aa400dbc8ac1c0543 (table test-table [id=cfe987178d114473b0070ec719853886]) successfully deleted
I20260619 18:50:11.864391 18426 ts_tablet_manager.cc:1918] T c5e56d3f59e1427aa400dbc8ac1c0543 P 8d3f261588f14e158cc4b8a3388f5136: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:50:11.864562 18426 log.cc:1196] T c5e56d3f59e1427aa400dbc8ac1c0543 P 8d3f261588f14e158cc4b8a3388f5136: Deleting WAL directory at /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-0/wal/wals/c5e56d3f59e1427aa400dbc8ac1c0543
I20260619 18:50:11.864954 18426 ts_tablet_manager.cc:1939] T c5e56d3f59e1427aa400dbc8ac1c0543 P 8d3f261588f14e158cc4b8a3388f5136: Deleting consensus metadata
Jun 19 18:50:11 dist-test-slave-m9wg krb5kdc[17883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895009, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.65@KRBTEST.COM
I20260619 18:50:11.866142 17914 catalog_manager.cc:4841] TS 8d3f261588f14e158cc4b8a3388f5136 (127.3.243.65:46855): tablet c5e56d3f59e1427aa400dbc8ac1c0543 (table test-table [id=cfe987178d114473b0070ec719853886]) successfully deleted
I20260619 18:50:11.872464 18399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:11.831154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36828 (local address 127.3.243.65:46855)
0619 18:50:11.831319 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 18:50:11.831325 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:50:11.852925 (+ 21600us) server_negotiation.cc:407] Connection header received
0619 18:50:11.853113 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:50:11.853119 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:50:11.853210 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:50:11.853317 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:50:11.857138 (+ 3821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:11.858332 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:11.860377 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:11.861111 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:11.861211 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:11.866866 (+ 5655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:50:11.866876 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:50:11.866888 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:50:11.866920 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:50:11.869868 (+ 2948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:11.870501 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:11.870507 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:11.870509 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:11.870649 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:11.871008 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:11.871011 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:11.871013 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:11.871332 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:50:11.871625 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0619 18:50:11.871924 (+ 299us) server_negotiation.cc:299] Negotiation successful
0619 18:50:11.872167 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 18:50:11.873127 18112 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:50:11.969599 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 17989
I20260619 18:50:11.984989 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 18123
Jun 19 18:50:11 dist-test-slave-m9wg krb5kdc[17883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895009, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.66@KRBTEST.COM
I20260619 18:50:11.998108 18398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:11.985638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33420 (local address 127.3.243.66:46229)
0619 18:50:11.985790 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 18:50:11.985794 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:50:11.985841 (+ 47us) server_negotiation.cc:407] Connection header received
0619 18:50:11.985985 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:50:11.985989 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:50:11.986050 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:50:11.986129 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:50:11.987263 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:11.987991 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:11.989225 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:11.989588 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:11.989677 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:11.993373 (+ 3696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:50:11.993386 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:50:11.993399 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:50:11.993433 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:50:11.995856 (+ 2423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:11.996465 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:11.996472 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:11.996474 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:11.996587 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:11.996879 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:11.996882 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:11.996884 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:11.997247 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:50:11.997456 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0619 18:50:11.997675 (+ 219us) server_negotiation.cc:299] Negotiation successful
0619 18:50:11.997817 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 18:50:11.998685 18246 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:50:12.097043 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 18123
I20260619 18:50:12.110394 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 18256
Jun 19 18:50:12 dist-test-slave-m9wg krb5kdc[17883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895009, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.67@KRBTEST.COM
I20260619 18:50:12.124199 18400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:12.111156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33652 (local address 127.3.243.67:34767)
0619 18:50:12.111317 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 18:50:12.111321 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:50:12.111338 (+ 17us) server_negotiation.cc:407] Connection header received
0619 18:50:12.111435 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:50:12.111439 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:50:12.111507 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:50:12.111594 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:50:12.112897 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:12.113675 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:12.114827 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:12.115207 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:12.115290 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:12.119478 (+ 4188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:50:12.119491 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:50:12.119513 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:50:12.119543 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:50:12.122043 (+ 2500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:12.122668 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:12.122674 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:12.122676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:12.122798 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:12.123054 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:12.123056 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:12.123058 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:12.123366 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:50:12.123576 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0619 18:50:12.123776 (+ 200us) server_negotiation.cc:299] Negotiation successful
0619 18:50:12.123923 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 18:50:12.124768 18379 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:50:12.210036 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 18256
I20260619 18:50:12.223006 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 17896
I20260619 18:50:12.224047 17958 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:50:12.329237 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 17896
I20260619 18:50:12.345906 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.126:39697
--webserver_interface=127.3.243.126
--webserver_port=39799
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab
--principal=kudu/127.3.243.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:46639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.126:39697
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:50:12.658773 18434 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:50:12.659111 18434 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:50:12.659207 18434 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:50:12.666345 18434 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:50:12.666474 18434 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:50:12.666527 18434 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:50:12.666582 18434 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:50:12.666623 18434 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:50:12.676538 18434 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:46639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.3.243.126:39697
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab
--principal=kudu/127.3.243.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.126:39697
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.243.126
--webserver_port=39799
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:50:12.677785 18434 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:50:12.678988 18434 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:50:12.699515 18443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:50:12.699460 18440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:50:12.699533 18441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:50:12.700385 18434 server_base.cc:1034] running on GCE node
I20260619 18:50:12.701207 18434 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:50:12.702718 18434 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:50:12.703909 18434 hybrid_clock.cc:648] HybridClock initialized: now 1781895012703865 us; error 34 us; skew 500 ppm
I20260619 18:50:12.704340 18434 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:50:12 dist-test-slave-m9wg krb5kdc[17883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895012, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:50:12.707662 18434 init.cc:377] Logged in from keytab as kudu/127.3.243.126@KRBTEST.COM (short username kudu)
I20260619 18:50:12.709203 18434 webserver.cc:460] Webserver started at http://127.3.243.126:39799/ using document root <none> and password file <none>
I20260619 18:50:12.709908 18434 fs_manager.cc:359] Metadata directory not provided
I20260619 18:50:12.710039 18434 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:50:12.715754 18434 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260619 18:50:12.718482 18450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:50:12.719689 18434 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 18:50:12.719843 18434 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/master-0/wal
uuid: "22ae91d05d6a4613906a3966068df987"
format_stamp: "Formatted at 2026-06-19 18:50:09 on dist-test-slave-m9wg"
server_key: "d2bfefb082a12cc1c57cf924a95d3392"
server_key_iv: "e8a328116c7198cab94c0c4d14d33ba9"
server_key_version: "encryptionkey@0"
I20260619 18:50:12.720906 18434 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:50:12.742990 18434 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:50:12.747687 18434 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:50:12.747923 18434 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:50:12.764882 18434 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.126:39697
I20260619 18:50:12.764894 18501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.126:39697 every 8 connection(s)
I20260619 18:50:12.766268 18434 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
I20260619 18:50:12.772940 18502 sys_catalog.cc:263] Verifying existing consensus state
I20260619 18:50:12.774775 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 18434
I20260619 18:50:12.775543 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.65:46855
--local_ip_for_outbound_sockets=127.3.243.65
--tserver_master_addrs=127.3.243.126:39697
--webserver_port=34531
--webserver_interface=127.3.243.65
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.65.keytab
--principal=kudu/127.3.243.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:46639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 18:50:12.776116 18502 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22ae91d05d6a4613906a3966068df987: Bootstrap starting.
I20260619 18:50:12.798512 18502 log.cc:824] T 00000000000000000000000000000000 P 22ae91d05d6a4613906a3966068df987: Log is configured to *not* fsync() on all Append() calls
I20260619 18:50:12.810160 18502 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22ae91d05d6a4613906a3966068df987: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 18:50:12.810642 18502 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22ae91d05d6a4613906a3966068df987: Bootstrap complete.
I20260619 18:50:12.819291 18502 raft_consensus.cc:348] T 00000000000000000000000000000000 P 22ae91d05d6a4613906a3966068df987 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22ae91d05d6a4613906a3966068df987" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 39697 } }
I20260619 18:50:12.820063 18502 raft_consensus.cc:729] T 00000000000000000000000000000000 P 22ae91d05d6a4613906a3966068df987 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22ae91d05d6a4613906a3966068df987, State: Initialized, Role: FOLLOWER
I20260619 18:50:12.820679 18502 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22ae91d05d6a4613906a3966068df987 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22ae91d05d6a4613906a3966068df987" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 39697 } }
I20260619 18:50:12.820900 18502 raft_consensus.cc:388] T 00000000000000000000000000000000 P 22ae91d05d6a4613906a3966068df987 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:50:12.821007 18502 raft_consensus.cc:482] T 00000000000000000000000000000000 P 22ae91d05d6a4613906a3966068df987 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:50:12.821157 18502 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 22ae91d05d6a4613906a3966068df987 [term 1 FOLLOWER]: Advancing to term 2
I20260619 18:50:12.824877 18502 raft_consensus.cc:504] T 00000000000000000000000000000000 P 22ae91d05d6a4613906a3966068df987 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22ae91d05d6a4613906a3966068df987" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 39697 } }
I20260619 18:50:12.825392 18502 leader_election.cc:302] T 00000000000000000000000000000000 P 22ae91d05d6a4613906a3966068df987 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22ae91d05d6a4613906a3966068df987; no voters:
I20260619 18:50:12.825961 18502 leader_election.cc:288] T 00000000000000000000000000000000 P 22ae91d05d6a4613906a3966068df987 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 18:50:12.826088 18506 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 22ae91d05d6a4613906a3966068df987 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 18:50:12.827132 18506 raft_consensus.cc:686] T 00000000000000000000000000000000 P 22ae91d05d6a4613906a3966068df987 [term 2 LEADER]: Becoming Leader. State: Replica: 22ae91d05d6a4613906a3966068df987, State: Running, Role: LEADER
I20260619 18:50:12.827677 18506 consensus_queue.cc:237] T 00000000000000000000000000000000 P 22ae91d05d6a4613906a3966068df987 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22ae91d05d6a4613906a3966068df987" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 39697 } }
I20260619 18:50:12.828138 18502 sys_catalog.cc:564] T 00000000000000000000000000000000 P 22ae91d05d6a4613906a3966068df987 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:50:12.830061 18507 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22ae91d05d6a4613906a3966068df987 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "22ae91d05d6a4613906a3966068df987" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22ae91d05d6a4613906a3966068df987" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 39697 } } }
I20260619 18:50:12.830416 18507 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22ae91d05d6a4613906a3966068df987 [sys.catalog]: This master's current role is: LEADER
I20260619 18:50:12.834105 18508 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22ae91d05d6a4613906a3966068df987 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 22ae91d05d6a4613906a3966068df987. Latest consensus state: current_term: 2 leader_uuid: "22ae91d05d6a4613906a3966068df987" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22ae91d05d6a4613906a3966068df987" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 39697 } } }
I20260619 18:50:12.834468 18508 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22ae91d05d6a4613906a3966068df987 [sys.catalog]: This master's current role is: LEADER
I20260619 18:50:12.835950 18511 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:50:12.844487 18511 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:50:12.845916 18511 catalog_manager.cc:1259] Loaded cluster ID: 13dc07384b5e42a49cc7067630c5ac3e
I20260619 18:50:12.846063 18511 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:50:12.849223 18511 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260619 18:50:12.855072 18523 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 18:50:12.862457 18511 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260619 18:50:12.868739 18511 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:50:12.869903 18511 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 22ae91d05d6a4613906a3966068df987: Loaded TSK: 0
I20260619 18:50:12.871025 18511 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 18:50:13.143332 18504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:50:13.143591 18504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:50:13.143723 18504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:50:13.150641 18504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:50:13.150749 18504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:50:13.150897 18504 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.65
I20260619 18:50:13.159997 18504 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:46639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.65.keytab
--principal=kudu/127.3.243.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.65:46855
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.243.65
--webserver_port=34531
--tserver_master_addrs=127.3.243.126:39697
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.65
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:50:13.161137 18504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:50:13.162374 18504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:50:13.175872 18530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:50:13.175868 18529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:50:13.176888 18532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:50:13.177136 18504 server_base.cc:1034] running on GCE node
I20260619 18:50:13.177724 18504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:50:13.178619 18504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:50:13.179808 18504 hybrid_clock.cc:648] HybridClock initialized: now 1781895013179754 us; error 50 us; skew 500 ppm
I20260619 18:50:13.180092 18504 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:50:13 dist-test-slave-m9wg krb5kdc[17883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895013, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:50:13.187995 18504 init.cc:377] Logged in from keytab as kudu/127.3.243.65@KRBTEST.COM (short username kudu)
I20260619 18:50:13.189373 18504 webserver.cc:460] Webserver started at http://127.3.243.65:34531/ using document root <none> and password file <none>
I20260619 18:50:13.190001 18504 fs_manager.cc:359] Metadata directory not provided
I20260619 18:50:13.190114 18504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:50:13.195317 18504 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260619 18:50:13.198047 18539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:50:13.199390 18504 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 18:50:13.199544 18504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-0/wal
uuid: "8d3f261588f14e158cc4b8a3388f5136"
format_stamp: "Formatted at 2026-06-19 18:50:10 on dist-test-slave-m9wg"
server_key: "4512017085bc64da101ab267e0829c0a"
server_key_iv: "f9aa9a356b97e4913808d79ae42c36f9"
server_key_version: "encryptionkey@0"
I20260619 18:50:13.200500 18504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:50:13.228956 18504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:50:13.232991 18504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:50:13.233222 18504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:50:13.234127 18504 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:50:13.235384 18504 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:50:13.235503 18504 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:50:13.235600 18504 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:50:13.235664 18504 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:50:13.264070 18504 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.65:46855
I20260619 18:50:13.264128 18651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.65:46855 every 8 connection(s)
I20260619 18:50:13.265439 18504 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-0/data/info.pb
I20260619 18:50:13.269563 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 18504
I20260619 18:50:13.270154 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.66:46229
--local_ip_for_outbound_sockets=127.3.243.66
--tserver_master_addrs=127.3.243.126:39697
--webserver_port=45245
--webserver_interface=127.3.243.66
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.66.keytab
--principal=kudu/127.3.243.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:46639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 18:50:13 dist-test-slave-m9wg krb5kdc[17883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895013, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.65@KRBTEST.COM for kudu/127.3.243.126@KRBTEST.COM
I20260619 18:50:13.287977 18654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:13.267979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.65:54577 (local address 127.3.243.126:39697)
0619 18:50:13.268568 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0619 18:50:13.268580 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 18:50:13.269169 (+ 589us) server_negotiation.cc:407] Connection header received
0619 18:50:13.271374 (+ 2205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:50:13.271393 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:50:13.271841 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:50:13.272252 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:50:13.273627 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:13.275382 (+ 1755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:13.276905 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:13.277582 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:13.277757 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:13.281042 (+ 3285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:50:13.281049 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:50:13.281064 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:50:13.281106 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:50:13.283702 (+ 2596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:13.284445 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:13.284451 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:13.284456 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:13.284620 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:13.285042 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:13.285047 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:13.285050 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:13.285537 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:50:13.285716 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0619 18:50:13.286449 (+ 733us) server_negotiation.cc:299] Negotiation successful
0619 18:50:13.286823 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":268,"threads_started":1}
I20260619 18:50:13.287978 18655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:13.268197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:39697 (local address 127.3.243.65:54577)
0619 18:50:13.268970 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:13.269031 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0619 18:50:13.270780 (+ 1749us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:50:13.272363 (+ 1583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:50:13.272371 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:50:13.272868 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:50:13.273407 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:50:13.273425 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:50:13.275525 (+ 2100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:50:13.275528 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:50:13.276769 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:50:13.276790 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:50:13.277803 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:50:13.277810 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:50:13.278063 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:13.278673 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:50:13.278694 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:50:13.280782 (+ 2088us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:50:13.283833 (+ 3051us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:50:13.283842 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:50:13.283857 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:50:13.284310 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:50:13.284738 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:50:13.284743 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:50:13.284747 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:50:13.284930 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:50:13.285661 (+ 731us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:50:13.285669 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:50:13.285979 (+ 310us) client_negotiation.cc:769] Sending connection context
0619 18:50:13.286199 (+ 220us) client_negotiation.cc:240] Negotiation successful
0619 18:50:13.286704 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":305,"threads_started":1}
I20260619 18:50:13.291030 18652 heartbeater.cc:347] Connected to a master server at 127.3.243.126:39697
I20260619 18:50:13.291364 18652 heartbeater.cc:464] Registering TS with master...
I20260619 18:50:13.292202 18652 heartbeater.cc:511] Master 127.3.243.126:39697 requested a full tablet report, sending...
I20260619 18:50:13.294508 18467 ts_manager.cc:194] Registered new tserver with Master: 8d3f261588f14e158cc4b8a3388f5136 (127.3.243.65:46855)
I20260619 18:50:13.296550 18467 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.65@KRBTEST.COM'} at 127.3.243.65:54577
W20260619 18:50:13.575495 18656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:50:13.575745 18656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:50:13.575837 18656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:50:13.582721 18656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:50:13.582823 18656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:50:13.582935 18656 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.66
I20260619 18:50:13.592001 18656 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:46639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.66.keytab
--principal=kudu/127.3.243.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.66:46229
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.243.66
--webserver_port=45245
--tserver_master_addrs=127.3.243.126:39697
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.66
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:50:13.593129 18656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:50:13.594290 18656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:50:13.606110 18661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:50:13.606155 18662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:50:13.606470 18664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:50:13.606774 18656 server_base.cc:1034] running on GCE node
I20260619 18:50:13.607322 18656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:50:13.608197 18656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:50:13.609392 18656 hybrid_clock.cc:648] HybridClock initialized: now 1781895013609344 us; error 42 us; skew 500 ppm
I20260619 18:50:13.609699 18656 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:50:13 dist-test-slave-m9wg krb5kdc[17883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895013, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:50:13.612566 18656 init.cc:377] Logged in from keytab as kudu/127.3.243.66@KRBTEST.COM (short username kudu)
I20260619 18:50:13.613785 18656 webserver.cc:460] Webserver started at http://127.3.243.66:45245/ using document root <none> and password file <none>
I20260619 18:50:13.614423 18656 fs_manager.cc:359] Metadata directory not provided
I20260619 18:50:13.614526 18656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:50:13.619392 18656 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.001s
I20260619 18:50:13.621923 18671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:50:13.623106 18656 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 18:50:13.623242 18656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-1/wal
uuid: "fa3486d01e9045dcba230304040c37f1"
format_stamp: "Formatted at 2026-06-19 18:50:10 on dist-test-slave-m9wg"
server_key: "6c3293924cc918b396555e43d5f1d38d"
server_key_iv: "9838ff3f14c6f992483f7f08be0ff4a7"
server_key_version: "encryptionkey@0"
I20260619 18:50:13.624205 18656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:50:13.647092 18656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:50:13.650591 18656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:50:13.650794 18656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:50:13.651654 18656 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:50:13.652882 18656 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:50:13.652973 18656 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:50:13.653048 18656 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:50:13.653098 18656 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:50:13.681775 18656 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.66:46229
I20260619 18:50:13.681839 18783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.66:46229 every 8 connection(s)
I20260619 18:50:13.683115 18656 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-1/data/info.pb
I20260619 18:50:13.691159 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 18656
I20260619 18:50:13.691870 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.67:34767
--local_ip_for_outbound_sockets=127.3.243.67
--tserver_master_addrs=127.3.243.126:39697
--webserver_port=38533
--webserver_interface=127.3.243.67
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.67.keytab
--principal=kudu/127.3.243.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:46639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 18:50:13 dist-test-slave-m9wg krb5kdc[17883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895013, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.66@KRBTEST.COM for kudu/127.3.243.126@KRBTEST.COM
I20260619 18:50:13.702665 18654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:13.685496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.66:41009 (local address 127.3.243.126:39697)
0619 18:50:13.685691 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 18:50:13.685697 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:50:13.686775 (+ 1078us) server_negotiation.cc:407] Connection header received
0619 18:50:13.688570 (+ 1795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:50:13.688574 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:50:13.688643 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:50:13.688726 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:50:13.690114 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:13.690795 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:13.693006 (+ 2211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:13.693434 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:13.693528 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:13.697444 (+ 3916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:50:13.697448 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:50:13.697452 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:50:13.697480 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:50:13.699829 (+ 2349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:13.700677 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:13.700680 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:13.700682 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:13.700800 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:13.701194 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:13.701197 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:13.701199 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:13.701435 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:50:13.701539 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 18:50:13.702260 (+ 721us) server_negotiation.cc:299] Negotiation successful
0619 18:50:13.702428 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 18:50:13.703917 18786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:13.685954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:39697 (local address 127.3.243.66:41009)
0619 18:50:13.686619 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:13.686655 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 18:50:13.688329 (+ 1674us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:50:13.688945 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:50:13.688953 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:50:13.689434 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:50:13.689967 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:50:13.689984 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:50:13.690968 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:50:13.690974 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:50:13.692856 (+ 1882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:50:13.692873 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:50:13.693636 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:50:13.693642 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:50:13.693888 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:13.694524 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:50:13.694550 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:50:13.697246 (+ 2696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:50:13.699958 (+ 2712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:50:13.699967 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:50:13.699982 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:50:13.700568 (+ 586us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:50:13.700897 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:50:13.700902 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:50:13.700905 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:50:13.701108 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:50:13.701537 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:50:13.701545 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:50:13.701921 (+ 376us) client_negotiation.cc:769] Sending connection context
0619 18:50:13.702177 (+ 256us) client_negotiation.cc:240] Negotiation successful
0619 18:50:13.702647 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":252,"threads_started":1}
I20260619 18:50:13.705314 18784 heartbeater.cc:347] Connected to a master server at 127.3.243.126:39697
I20260619 18:50:13.705585 18784 heartbeater.cc:464] Registering TS with master...
I20260619 18:50:13.706220 18784 heartbeater.cc:511] Master 127.3.243.126:39697 requested a full tablet report, sending...
I20260619 18:50:13.707537 18467 ts_manager.cc:194] Registered new tserver with Master: fa3486d01e9045dcba230304040c37f1 (127.3.243.66:46229)
I20260619 18:50:13.708570 18467 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.66@KRBTEST.COM'} at 127.3.243.66:41009
W20260619 18:50:13.997437 18787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:50:13.997675 18787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:50:13.997769 18787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:50:14.004659 18787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:50:14.004755 18787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:50:14.004858 18787 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.67
I20260619 18:50:14.013854 18787 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:46639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.67.keytab
--principal=kudu/127.3.243.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.67:34767
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.243.67
--webserver_port=38533
--tserver_master_addrs=127.3.243.126:39697
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.67
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:50:14.014952 18787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:50:14.016103 18787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:50:14.027731 18792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:50:14.027817 18795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:50:14.027823 18793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:50:14.028939 18787 server_base.cc:1034] running on GCE node
I20260619 18:50:14.029501 18787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:50:14.030300 18787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:50:14.031514 18787 hybrid_clock.cc:648] HybridClock initialized: now 1781895014031453 us; error 44 us; skew 500 ppm
I20260619 18:50:14.031800 18787 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:50:14 dist-test-slave-m9wg krb5kdc[17883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895014, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:50:14.034643 18787 init.cc:377] Logged in from keytab as kudu/127.3.243.67@KRBTEST.COM (short username kudu)
I20260619 18:50:14.035959 18787 webserver.cc:460] Webserver started at http://127.3.243.67:38533/ using document root <none> and password file <none>
I20260619 18:50:14.036607 18787 fs_manager.cc:359] Metadata directory not provided
I20260619 18:50:14.036713 18787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:50:14.041448 18787 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.004s
I20260619 18:50:14.043864 18802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:50:14.045071 18787 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 18:50:14.045192 18787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-2/wal
uuid: "a87fb8ce1b304e8e9dbc3e969e690d9b"
format_stamp: "Formatted at 2026-06-19 18:50:11 on dist-test-slave-m9wg"
server_key: "38f5df90f25296131d3953cc271cb82b"
server_key_iv: "e66a5563157d00381fb091c0243bea8d"
server_key_version: "encryptionkey@0"
I20260619 18:50:14.046047 18787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:50:14.061797 18787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:50:14.065486 18787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:50:14.065688 18787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:50:14.066519 18787 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:50:14.067667 18787 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:50:14.067755 18787 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:50:14.067826 18787 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:50:14.067880 18787 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:50:14.096675 18787 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.67:34767
I20260619 18:50:14.096725 18914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.67:34767 every 8 connection(s)
I20260619 18:50:14.098007 18787 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-2/data/info.pb
I20260619 18:50:14.104488 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 18787
Jun 19 18:50:14 dist-test-slave-m9wg krb5kdc[17883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895014, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.67@KRBTEST.COM for kudu/127.3.243.126@KRBTEST.COM
I20260619 18:50:14.117185 18654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:14.100301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.67:42359 (local address 127.3.243.126:39697)
0619 18:50:14.100468 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 18:50:14.100473 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:50:14.101560 (+ 1087us) server_negotiation.cc:407] Connection header received
0619 18:50:14.103305 (+ 1745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:50:14.103310 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:50:14.103387 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:50:14.103470 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:50:14.104985 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:14.105905 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:14.107410 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:14.107888 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:14.108011 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:14.111257 (+ 3246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:50:14.111261 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:50:14.111264 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:50:14.111292 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:50:14.113926 (+ 2634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:14.114750 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:14.114754 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:14.114756 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:14.114916 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:14.115399 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:14.115402 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:14.115404 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:14.115694 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:50:14.115813 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 18:50:14.116661 (+ 848us) server_negotiation.cc:299] Negotiation successful
0619 18:50:14.116855 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 18:50:14.118075 18917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:14.100747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:39697 (local address 127.3.243.67:42359)
0619 18:50:14.101395 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:14.101436 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 18:50:14.103017 (+ 1581us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:50:14.103711 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:50:14.103719 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:50:14.104241 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:50:14.104823 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:50:14.104841 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:50:14.106060 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:50:14.106064 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:50:14.107267 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:50:14.107280 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:50:14.108101 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:50:14.108109 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:50:14.108368 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:14.108952 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:50:14.108973 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:50:14.111057 (+ 2084us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:50:14.114045 (+ 2988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:50:14.114054 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:50:14.114071 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:50:14.114627 (+ 556us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:50:14.115038 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:50:14.115042 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:50:14.115046 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:50:14.115280 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:50:14.115819 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:50:14.115826 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:50:14.116247 (+ 421us) client_negotiation.cc:769] Sending connection context
0619 18:50:14.116536 (+ 289us) client_negotiation.cc:240] Negotiation successful
0619 18:50:14.116890 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":258,"threads_started":1}
I20260619 18:50:14.119159 18918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:14.105324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54858 (local address 127.3.243.126:39697)
0619 18:50:14.105764 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0619 18:50:14.105769 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:50:14.106088 (+ 319us) server_negotiation.cc:407] Connection header received
0619 18:50:14.106246 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:50:14.106251 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:50:14.106308 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:50:14.106405 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:50:14.107790 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:14.108590 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:14.109871 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:14.110286 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:14.110388 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:14.113183 (+ 2795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:50:14.113187 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:50:14.113191 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:50:14.113228 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:50:14.116373 (+ 3145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:14.117137 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:14.117141 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:14.117144 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:14.117379 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:14.117721 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:14.117726 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:14.117730 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:14.118088 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:50:14.118324 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0619 18:50:14.118609 (+ 285us) server_negotiation.cc:299] Negotiation successful
0619 18:50:14.118818 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":211,"threads_started":1}
I20260619 18:50:14.119532 18915 heartbeater.cc:347] Connected to a master server at 127.3.243.126:39697
I20260619 18:50:14.119822 18915 heartbeater.cc:464] Registering TS with master...
I20260619 18:50:14.120580 18915 heartbeater.cc:511] Master 127.3.243.126:39697 requested a full tablet report, sending...
I20260619 18:50:14.121886 18467 ts_manager.cc:194] Registered new tserver with Master: a87fb8ce1b304e8e9dbc3e969e690d9b (127.3.243.67:34767)
I20260619 18:50:14.122627 18467 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.67@KRBTEST.COM'} at 127.3.243.67:42359
I20260619 18:50:14.133186 4045 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 18:50:14.139016 18918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:14.133984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54860 (local address 127.3.243.126:39697)
0619 18:50:14.134134 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 18:50:14.134138 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:50:14.134585 (+ 447us) server_negotiation.cc:407] Connection header received
0619 18:50:14.134736 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:50:14.134739 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:50:14.134860 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:50:14.134963 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:50:14.135259 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:14.135888 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:14.137333 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:14.137709 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:14.137788 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:14.138169 (+ 381us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:50:14.138609 (+ 440us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:50:14.138703 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 18:50:14.138796 (+ 93us) server_negotiation.cc:299] Negotiation successful
0619 18:50:14.138849 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 18:50:14.139741 18467 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:54860:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:50:14.146461 18467 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:50:14.167740 18926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:14.157119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36834 (local address 127.3.243.65:46855)
0619 18:50:14.159422 (+ 2303us) server_negotiation.cc:206] Beginning negotiation
0619 18:50:14.159430 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:50:14.159456 (+ 26us) server_negotiation.cc:407] Connection header received
0619 18:50:14.159546 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:50:14.159554 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:50:14.159801 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:50:14.159988 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:50:14.161627 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:14.163222 (+ 1595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:14.165639 (+ 2417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:14.166778 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:14.166928 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:14.167119 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0619 18:50:14.167273 (+ 154us) server_negotiation.cc:299] Negotiation successful
0619 18:50:14.167429 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2145,"thread_start_us":2024,"threads_started":1}
I20260619 18:50:14.167780 18927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:14.157006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.65:46855 (local address 127.0.0.1:36834)
0619 18:50:14.158661 (+ 1655us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:14.158702 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 18:50:14.158872 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:50:14.161020 (+ 2148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:50:14.161025 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:50:14.161053 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:50:14.161440 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:50:14.161455 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:50:14.163413 (+ 1958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:50:14.163417 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:50:14.165493 (+ 2076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:50:14.165505 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:50:14.166949 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:50:14.166953 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:50:14.167072 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:14.167105 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 18:50:14.167209 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 18:50:14.167386 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1376,"mutex_wait_us":91,"thread_start_us":3786,"threads_started":1}
I20260619 18:50:14.168797 18925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:14.159061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.67:34767 (local address 127.0.0.1:33664)
0619 18:50:14.161070 (+ 2009us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:14.161086 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 18:50:14.161268 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:50:14.161797 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:50:14.161801 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:50:14.161822 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:50:14.162139 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:50:14.162148 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:50:14.163394 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:50:14.163399 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:50:14.166091 (+ 2692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:50:14.166105 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:50:14.167998 (+ 1893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:50:14.168005 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:50:14.168190 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:14.168223 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 18:50:14.168304 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 18:50:14.168430 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1885,"mutex_wait_us":184,"thread_start_us":391,"threads_started":1}
I20260619 18:50:14.168809 18930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:14.159513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33664 (local address 127.3.243.67:34767)
0619 18:50:14.160633 (+ 1120us) server_negotiation.cc:206] Beginning negotiation
0619 18:50:14.160639 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:50:14.161172 (+ 533us) server_negotiation.cc:407] Connection header received
0619 18:50:14.161441 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:50:14.161448 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:50:14.161647 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:50:14.161789 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:50:14.162276 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:14.163242 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:14.166244 (+ 3002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:14.167531 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:14.167648 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:14.167846 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0619 18:50:14.168383 (+ 537us) server_negotiation.cc:299] Negotiation successful
0619 18:50:14.168539 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":980,"thread_start_us":828,"threads_started":1}
I20260619 18:50:14.170413 18929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:14.157107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.66:46229 (local address 127.0.0.1:33422)
0619 18:50:14.159692 (+ 2585us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:14.159709 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 18:50:14.159824 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:50:14.160921 (+ 1097us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:50:14.160928 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:50:14.160961 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:50:14.161414 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:50:14.161431 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:50:14.163361 (+ 1930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:50:14.163366 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:50:14.166106 (+ 2740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:50:14.166115 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:50:14.169659 (+ 3544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:50:14.169664 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:50:14.169834 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:14.169863 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 18:50:14.169950 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 18:50:14.170067 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2472,"mutex_wait_us":200,"thread_start_us":1126,"threads_started":1}
I20260619 18:50:14.170838 18928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:14.157774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33422 (local address 127.3.243.66:46229)
0619 18:50:14.160314 (+ 2540us) server_negotiation.cc:206] Beginning negotiation
0619 18:50:14.160321 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:50:14.160355 (+ 34us) server_negotiation.cc:407] Connection header received
0619 18:50:14.160455 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:50:14.160461 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:50:14.160678 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:50:14.161461 (+ 783us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:50:14.161732 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:14.163214 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:14.167208 (+ 3994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:14.169471 (+ 2263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:14.169582 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:14.169782 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0619 18:50:14.170002 (+ 220us) server_negotiation.cc:299] Negotiation successful
0619 18:50:14.170157 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2390,"thread_start_us":2244,"threads_started":1}
I20260619 18:50:14.171975 18587 tablet_service.cc:1467] Processing CreateTablet for tablet 485ff9db72c14d9ca26b2a9b2e921d50 (DEFAULT_TABLE table=test-table [id=4a76f690ffe14db98311da09e793ba0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:50:14.172299 18850 tablet_service.cc:1467] Processing CreateTablet for tablet 485ff9db72c14d9ca26b2a9b2e921d50 (DEFAULT_TABLE table=test-table [id=4a76f690ffe14db98311da09e793ba0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:50:14.173489 18850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 485ff9db72c14d9ca26b2a9b2e921d50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:50:14.173504 18587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 485ff9db72c14d9ca26b2a9b2e921d50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:50:14.173903 18719 tablet_service.cc:1467] Processing CreateTablet for tablet 485ff9db72c14d9ca26b2a9b2e921d50 (DEFAULT_TABLE table=test-table [id=4a76f690ffe14db98311da09e793ba0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:50:14.174935 18719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 485ff9db72c14d9ca26b2a9b2e921d50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:50:14.187103 18931 tablet_bootstrap.cc:492] T 485ff9db72c14d9ca26b2a9b2e921d50 P fa3486d01e9045dcba230304040c37f1: Bootstrap starting.
I20260619 18:50:14.188261 18932 tablet_bootstrap.cc:492] T 485ff9db72c14d9ca26b2a9b2e921d50 P 8d3f261588f14e158cc4b8a3388f5136: Bootstrap starting.
I20260619 18:50:14.190478 18931 tablet_bootstrap.cc:654] T 485ff9db72c14d9ca26b2a9b2e921d50 P fa3486d01e9045dcba230304040c37f1: Neither blocks nor log segments found. Creating new log.
I20260619 18:50:14.191725 18932 tablet_bootstrap.cc:654] T 485ff9db72c14d9ca26b2a9b2e921d50 P 8d3f261588f14e158cc4b8a3388f5136: Neither blocks nor log segments found. Creating new log.
I20260619 18:50:14.191736 18931 log.cc:824] T 485ff9db72c14d9ca26b2a9b2e921d50 P fa3486d01e9045dcba230304040c37f1: Log is configured to *not* fsync() on all Append() calls
I20260619 18:50:14.192986 18932 log.cc:824] T 485ff9db72c14d9ca26b2a9b2e921d50 P 8d3f261588f14e158cc4b8a3388f5136: Log is configured to *not* fsync() on all Append() calls
I20260619 18:50:14.194231 18933 tablet_bootstrap.cc:492] T 485ff9db72c14d9ca26b2a9b2e921d50 P a87fb8ce1b304e8e9dbc3e969e690d9b: Bootstrap starting.
I20260619 18:50:14.196095 18932 tablet_bootstrap.cc:492] T 485ff9db72c14d9ca26b2a9b2e921d50 P 8d3f261588f14e158cc4b8a3388f5136: No bootstrap required, opened a new log
I20260619 18:50:14.196420 18932 ts_tablet_manager.cc:1397] T 485ff9db72c14d9ca26b2a9b2e921d50 P 8d3f261588f14e158cc4b8a3388f5136: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.002s
I20260619 18:50:14.197707 18933 tablet_bootstrap.cc:654] T 485ff9db72c14d9ca26b2a9b2e921d50 P a87fb8ce1b304e8e9dbc3e969e690d9b: Neither blocks nor log segments found. Creating new log.
I20260619 18:50:14.198558 18933 log.cc:824] T 485ff9db72c14d9ca26b2a9b2e921d50 P a87fb8ce1b304e8e9dbc3e969e690d9b: Log is configured to *not* fsync() on all Append() calls
I20260619 18:50:14.200850 18931 tablet_bootstrap.cc:492] T 485ff9db72c14d9ca26b2a9b2e921d50 P fa3486d01e9045dcba230304040c37f1: No bootstrap required, opened a new log
I20260619 18:50:14.201081 18931 ts_tablet_manager.cc:1397] T 485ff9db72c14d9ca26b2a9b2e921d50 P fa3486d01e9045dcba230304040c37f1: Time spent bootstrapping tablet: real 0.014s user 0.011s sys 0.001s
I20260619 18:50:14.201756 18933 tablet_bootstrap.cc:492] T 485ff9db72c14d9ca26b2a9b2e921d50 P a87fb8ce1b304e8e9dbc3e969e690d9b: No bootstrap required, opened a new log
I20260619 18:50:14.201992 18933 ts_tablet_manager.cc:1397] T 485ff9db72c14d9ca26b2a9b2e921d50 P a87fb8ce1b304e8e9dbc3e969e690d9b: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260619 18:50:14.206137 18932 raft_consensus.cc:348] T 485ff9db72c14d9ca26b2a9b2e921d50 P 8d3f261588f14e158cc4b8a3388f5136 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d3f261588f14e158cc4b8a3388f5136" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 46855 } } peers { permanent_uuid: "fa3486d01e9045dcba230304040c37f1" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 46229 } } peers { permanent_uuid: "a87fb8ce1b304e8e9dbc3e969e690d9b" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 34767 } }
I20260619 18:50:14.206545 18932 raft_consensus.cc:374] T 485ff9db72c14d9ca26b2a9b2e921d50 P 8d3f261588f14e158cc4b8a3388f5136 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:50:14.206637 18932 raft_consensus.cc:729] T 485ff9db72c14d9ca26b2a9b2e921d50 P 8d3f261588f14e158cc4b8a3388f5136 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d3f261588f14e158cc4b8a3388f5136, State: Initialized, Role: FOLLOWER
I20260619 18:50:14.207192 18932 consensus_queue.cc:260] T 485ff9db72c14d9ca26b2a9b2e921d50 P 8d3f261588f14e158cc4b8a3388f5136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d3f261588f14e158cc4b8a3388f5136" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 46855 } } peers { permanent_uuid: "fa3486d01e9045dcba230304040c37f1" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 46229 } } peers { permanent_uuid: "a87fb8ce1b304e8e9dbc3e969e690d9b" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 34767 } }
I20260619 18:50:14.208271 18652 heartbeater.cc:503] Master 127.3.243.126:39697 was elected leader, sending a full tablet report...
I20260619 18:50:14.208879 18932 ts_tablet_manager.cc:1428] T 485ff9db72c14d9ca26b2a9b2e921d50 P 8d3f261588f14e158cc4b8a3388f5136: Time spent starting tablet: real 0.012s user 0.006s sys 0.006s
I20260619 18:50:14.208889 18931 raft_consensus.cc:348] T 485ff9db72c14d9ca26b2a9b2e921d50 P fa3486d01e9045dcba230304040c37f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d3f261588f14e158cc4b8a3388f5136" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 46855 } } peers { permanent_uuid: "fa3486d01e9045dcba230304040c37f1" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 46229 } } peers { permanent_uuid: "a87fb8ce1b304e8e9dbc3e969e690d9b" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 34767 } }
I20260619 18:50:14.209266 18931 raft_consensus.cc:374] T 485ff9db72c14d9ca26b2a9b2e921d50 P fa3486d01e9045dcba230304040c37f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:50:14.209354 18931 raft_consensus.cc:729] T 485ff9db72c14d9ca26b2a9b2e921d50 P fa3486d01e9045dcba230304040c37f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa3486d01e9045dcba230304040c37f1, State: Initialized, Role: FOLLOWER
I20260619 18:50:14.209872 18931 consensus_queue.cc:260] T 485ff9db72c14d9ca26b2a9b2e921d50 P fa3486d01e9045dcba230304040c37f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d3f261588f14e158cc4b8a3388f5136" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 46855 } } peers { permanent_uuid: "fa3486d01e9045dcba230304040c37f1" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 46229 } } peers { permanent_uuid: "a87fb8ce1b304e8e9dbc3e969e690d9b" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 34767 } }
I20260619 18:50:14.210821 18784 heartbeater.cc:503] Master 127.3.243.126:39697 was elected leader, sending a full tablet report...
I20260619 18:50:14.211459 18931 ts_tablet_manager.cc:1428] T 485ff9db72c14d9ca26b2a9b2e921d50 P fa3486d01e9045dcba230304040c37f1: Time spent starting tablet: real 0.010s user 0.007s sys 0.003s
I20260619 18:50:14.212124 18933 raft_consensus.cc:348] T 485ff9db72c14d9ca26b2a9b2e921d50 P a87fb8ce1b304e8e9dbc3e969e690d9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d3f261588f14e158cc4b8a3388f5136" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 46855 } } peers { permanent_uuid: "fa3486d01e9045dcba230304040c37f1" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 46229 } } peers { permanent_uuid: "a87fb8ce1b304e8e9dbc3e969e690d9b" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 34767 } }
I20260619 18:50:14.212450 18933 raft_consensus.cc:374] T 485ff9db72c14d9ca26b2a9b2e921d50 P a87fb8ce1b304e8e9dbc3e969e690d9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:50:14.212541 18933 raft_consensus.cc:729] T 485ff9db72c14d9ca26b2a9b2e921d50 P a87fb8ce1b304e8e9dbc3e969e690d9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a87fb8ce1b304e8e9dbc3e969e690d9b, State: Initialized, Role: FOLLOWER
I20260619 18:50:14.213074 18933 consensus_queue.cc:260] T 485ff9db72c14d9ca26b2a9b2e921d50 P a87fb8ce1b304e8e9dbc3e969e690d9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d3f261588f14e158cc4b8a3388f5136" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 46855 } } peers { permanent_uuid: "fa3486d01e9045dcba230304040c37f1" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 46229 } } peers { permanent_uuid: "a87fb8ce1b304e8e9dbc3e969e690d9b" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 34767 } }
I20260619 18:50:14.214093 18915 heartbeater.cc:503] Master 127.3.243.126:39697 was elected leader, sending a full tablet report...
I20260619 18:50:14.214640 18933 ts_tablet_manager.cc:1428] T 485ff9db72c14d9ca26b2a9b2e921d50 P a87fb8ce1b304e8e9dbc3e969e690d9b: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
W20260619 18:50:14.267277 18653 tablet.cc:2369] T 485ff9db72c14d9ca26b2a9b2e921d50 P 8d3f261588f14e158cc4b8a3388f5136: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 18:50:14.349436 18916 tablet.cc:2369] T 485ff9db72c14d9ca26b2a9b2e921d50 P a87fb8ce1b304e8e9dbc3e969e690d9b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:50:14.432868 18937 raft_consensus.cc:482] T 485ff9db72c14d9ca26b2a9b2e921d50 P 8d3f261588f14e158cc4b8a3388f5136 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:50:14.433163 18937 raft_consensus.cc:504] T 485ff9db72c14d9ca26b2a9b2e921d50 P 8d3f261588f14e158cc4b8a3388f5136 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d3f261588f14e158cc4b8a3388f5136" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 46855 } } peers { permanent_uuid: "fa3486d01e9045dcba230304040c37f1" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 46229 } } peers { permanent_uuid: "a87fb8ce1b304e8e9dbc3e969e690d9b" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 34767 } }
W20260619 18:50:14.434877 18785 tablet.cc:2369] T 485ff9db72c14d9ca26b2a9b2e921d50 P fa3486d01e9045dcba230304040c37f1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:50:14.434947 18937 leader_election.cc:288] T 485ff9db72c14d9ca26b2a9b2e921d50 P 8d3f261588f14e158cc4b8a3388f5136 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa3486d01e9045dcba230304040c37f1 (127.3.243.66:46229), a87fb8ce1b304e8e9dbc3e969e690d9b (127.3.243.67:34767)
I20260619 18:50:14.440990 18928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:14.435232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.65:50825 (local address 127.3.243.66:46229)
0619 18:50:14.435443 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0619 18:50:14.435449 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:50:14.435985 (+ 536us) server_negotiation.cc:407] Connection header received
0619 18:50:14.436209 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:50:14.436215 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:50:14.436292 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:50:14.436392 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:50:14.437046 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:14.437742 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:14.439410 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:14.440183 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:14.440323 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:14.440404 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 18:50:14.440647 (+ 243us) server_negotiation.cc:299] Negotiation successful
0619 18:50:14.440764 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 18:50:14.441079 18940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:14.435244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.66:46229 (local address 127.3.243.65:50825)
0619 18:50:14.435887 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:14.435904 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 18:50:14.436041 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:50:14.436566 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:50:14.436571 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:50:14.436599 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:50:14.436914 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:50:14.436924 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:50:14.437866 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:50:14.437871 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:50:14.439273 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:50:14.439284 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:50:14.440360 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:50:14.440366 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:50:14.440512 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:14.440534 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 18:50:14.440613 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 18:50:14.440738 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":343,"threads_started":1}
I20260619 18:50:14.441419 18930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:14.436249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.65:37573 (local address 127.3.243.67:34767)
0619 18:50:14.436450 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 18:50:14.436455 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:50:14.436760 (+ 305us) server_negotiation.cc:407] Connection header received
0619 18:50:14.436919 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:50:14.436923 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:50:14.436986 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:50:14.437055 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:50:14.437538 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:14.438226 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:14.439802 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:14.440599 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:14.440736 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:14.440805 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 18:50:14.441074 (+ 269us) server_negotiation.cc:299] Negotiation successful
0619 18:50:14.441193 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 18:50:14.441465 18941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:14.436030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.67:34767 (local address 127.3.243.65:37573)
0619 18:50:14.436652 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:14.436669 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 18:50:14.436794 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:50:14.437079 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:50:14.437082 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:50:14.437103 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:50:14.437394 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:50:14.437402 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:50:14.438346 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:50:14.438351 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:50:14.439662 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:50:14.439674 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:50:14.440795 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:50:14.440801 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:50:14.440936 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:14.440958 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 18:50:14.441031 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 18:50:14.441146 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":320,"threads_started":1}
I20260619 18:50:14.442066 18739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "485ff9db72c14d9ca26b2a9b2e921d50" candidate_uuid: "8d3f261588f14e158cc4b8a3388f5136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa3486d01e9045dcba230304040c37f1" is_pre_election: true
I20260619 18:50:14.442252 18870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "485ff9db72c14d9ca26b2a9b2e921d50" candidate_uuid: "8d3f261588f14e158cc4b8a3388f5136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a87fb8ce1b304e8e9dbc3e969e690d9b" is_pre_election: true
I20260619 18:50:14.442505 18739 raft_consensus.cc:2454] T 485ff9db72c14d9ca26b2a9b2e921d50 P fa3486d01e9045dcba230304040c37f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d3f261588f14e158cc4b8a3388f5136 in term 0.
I20260619 18:50:14.442728 18870 raft_consensus.cc:2454] T 485ff9db72c14d9ca26b2a9b2e921d50 P a87fb8ce1b304e8e9dbc3e969e690d9b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d3f261588f14e158cc4b8a3388f5136 in term 0.
I20260619 18:50:14.443104 18541 leader_election.cc:302] T 485ff9db72c14d9ca26b2a9b2e921d50 P 8d3f261588f14e158cc4b8a3388f5136 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d3f261588f14e158cc4b8a3388f5136, fa3486d01e9045dcba230304040c37f1; no voters:
I20260619 18:50:14.443432 18937 raft_consensus.cc:2781] T 485ff9db72c14d9ca26b2a9b2e921d50 P 8d3f261588f14e158cc4b8a3388f5136 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:50:14.443557 18937 raft_consensus.cc:482] T 485ff9db72c14d9ca26b2a9b2e921d50 P 8d3f261588f14e158cc4b8a3388f5136 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:50:14.443656 18937 raft_consensus.cc:3037] T 485ff9db72c14d9ca26b2a9b2e921d50 P 8d3f261588f14e158cc4b8a3388f5136 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:50:14.446369 18937 raft_consensus.cc:504] T 485ff9db72c14d9ca26b2a9b2e921d50 P 8d3f261588f14e158cc4b8a3388f5136 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d3f261588f14e158cc4b8a3388f5136" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 46855 } } peers { permanent_uuid: "fa3486d01e9045dcba230304040c37f1" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 46229 } } peers { permanent_uuid: "a87fb8ce1b304e8e9dbc3e969e690d9b" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 34767 } }
I20260619 18:50:14.446893 18937 leader_election.cc:288] T 485ff9db72c14d9ca26b2a9b2e921d50 P 8d3f261588f14e158cc4b8a3388f5136 [CANDIDATE]: Term 1 election: Requested vote from peers fa3486d01e9045dcba230304040c37f1 (127.3.243.66:46229), a87fb8ce1b304e8e9dbc3e969e690d9b (127.3.243.67:34767)
I20260619 18:50:14.447305 18739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "485ff9db72c14d9ca26b2a9b2e921d50" candidate_uuid: "8d3f261588f14e158cc4b8a3388f5136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa3486d01e9045dcba230304040c37f1"
I20260619 18:50:14.447351 18870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "485ff9db72c14d9ca26b2a9b2e921d50" candidate_uuid: "8d3f261588f14e158cc4b8a3388f5136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a87fb8ce1b304e8e9dbc3e969e690d9b"
I20260619 18:50:14.447474 18739 raft_consensus.cc:3037] T 485ff9db72c14d9ca26b2a9b2e921d50 P fa3486d01e9045dcba230304040c37f1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:50:14.447537 18870 raft_consensus.cc:3037] T 485ff9db72c14d9ca26b2a9b2e921d50 P a87fb8ce1b304e8e9dbc3e969e690d9b [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:50:14.449676 18739 raft_consensus.cc:2454] T 485ff9db72c14d9ca26b2a9b2e921d50 P fa3486d01e9045dcba230304040c37f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d3f261588f14e158cc4b8a3388f5136 in term 1.
I20260619 18:50:14.449807 18870 raft_consensus.cc:2454] T 485ff9db72c14d9ca26b2a9b2e921d50 P a87fb8ce1b304e8e9dbc3e969e690d9b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d3f261588f14e158cc4b8a3388f5136 in term 1.
I20260619 18:50:14.450019 18541 leader_election.cc:302] T 485ff9db72c14d9ca26b2a9b2e921d50 P 8d3f261588f14e158cc4b8a3388f5136 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d3f261588f14e158cc4b8a3388f5136, fa3486d01e9045dcba230304040c37f1; no voters:
I20260619 18:50:14.450286 18937 raft_consensus.cc:2781] T 485ff9db72c14d9ca26b2a9b2e921d50 P 8d3f261588f14e158cc4b8a3388f5136 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:50:14.450698 18937 raft_consensus.cc:686] T 485ff9db72c14d9ca26b2a9b2e921d50 P 8d3f261588f14e158cc4b8a3388f5136 [term 1 LEADER]: Becoming Leader. State: Replica: 8d3f261588f14e158cc4b8a3388f5136, State: Running, Role: LEADER
I20260619 18:50:14.451138 18937 consensus_queue.cc:237] T 485ff9db72c14d9ca26b2a9b2e921d50 P 8d3f261588f14e158cc4b8a3388f5136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d3f261588f14e158cc4b8a3388f5136" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 46855 } } peers { permanent_uuid: "fa3486d01e9045dcba230304040c37f1" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 46229 } } peers { permanent_uuid: "a87fb8ce1b304e8e9dbc3e969e690d9b" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 34767 } }
I20260619 18:50:14.455603 18467 catalog_manager.cc:5495] T 485ff9db72c14d9ca26b2a9b2e921d50 P 8d3f261588f14e158cc4b8a3388f5136 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d3f261588f14e158cc4b8a3388f5136 (127.3.243.65). New cstate: current_term: 1 leader_uuid: "8d3f261588f14e158cc4b8a3388f5136" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d3f261588f14e158cc4b8a3388f5136" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 46855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fa3486d01e9045dcba230304040c37f1" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 46229 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a87fb8ce1b304e8e9dbc3e969e690d9b" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 34767 } health_report { overall_health: UNKNOWN } } }
I20260619 18:50:14.527662 18926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:14.522703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36846 (local address 127.3.243.65:46855)
0619 18:50:14.522885 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 18:50:14.522891 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:50:14.522908 (+ 17us) server_negotiation.cc:407] Connection header received
0619 18:50:14.523082 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:50:14.523086 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:50:14.523160 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:50:14.523281 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:50:14.523643 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:14.524398 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:14.525757 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:14.526161 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:14.526253 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:14.526677 (+ 424us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:50:14.527187 (+ 510us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:50:14.527288 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 18:50:14.527354 (+ 66us) server_negotiation.cc:299] Negotiation successful
0619 18:50:14.527418 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 18:50:14.534994 18870 raft_consensus.cc:1261] T 485ff9db72c14d9ca26b2a9b2e921d50 P a87fb8ce1b304e8e9dbc3e969e690d9b [term 1 FOLLOWER]: Refusing update from remote peer 8d3f261588f14e158cc4b8a3388f5136: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:50:14.534997 18739 raft_consensus.cc:1261] T 485ff9db72c14d9ca26b2a9b2e921d50 P fa3486d01e9045dcba230304040c37f1 [term 1 FOLLOWER]: Refusing update from remote peer 8d3f261588f14e158cc4b8a3388f5136: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:50:14.535867 18942 consensus_queue.cc:1035] T 485ff9db72c14d9ca26b2a9b2e921d50 P 8d3f261588f14e158cc4b8a3388f5136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa3486d01e9045dcba230304040c37f1" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 46229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:50:14.536234 18937 consensus_queue.cc:1035] T 485ff9db72c14d9ca26b2a9b2e921d50 P 8d3f261588f14e158cc4b8a3388f5136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a87fb8ce1b304e8e9dbc3e969e690d9b" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 34767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:50:14.550662 18946 mvcc.cc:205] Tried to move back new op lower bound from 7298641979526529024 to 7298641979195240448. Current Snapshot: MvccSnapshot[applied={T|T < 7298641979526529024}]
I20260619 18:50:14.560314 18466 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:54860:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:50:14.560585 18466 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:54860:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:50:14.564128 18466 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 22ae91d05d6a4613906a3966068df987: Sending DeleteTablet for 3 replicas of tablet 485ff9db72c14d9ca26b2a9b2e921d50
I20260619 18:50:14.565275 18587 tablet_service.cc:1514] Processing DeleteTablet for tablet 485ff9db72c14d9ca26b2a9b2e921d50 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:50:14 UTC) from {username='kudu', principal='kudu/127.3.243.126@KRBTEST.COM'} at 127.0.0.1:36834
I20260619 18:50:14.565300 18850 tablet_service.cc:1514] Processing DeleteTablet for tablet 485ff9db72c14d9ca26b2a9b2e921d50 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:50:14 UTC) from {username='kudu', principal='kudu/127.3.243.126@KRBTEST.COM'} at 127.0.0.1:33664
I20260619 18:50:14.565310 18719 tablet_service.cc:1514] Processing DeleteTablet for tablet 485ff9db72c14d9ca26b2a9b2e921d50 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:50:14 UTC) from {username='kudu', principal='kudu/127.3.243.126@KRBTEST.COM'} at 127.0.0.1:33422
I20260619 18:50:14.569238 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 18504
I20260619 18:50:14.577715 18954 tablet_replica.cc:330] T 485ff9db72c14d9ca26b2a9b2e921d50 P a87fb8ce1b304e8e9dbc3e969e690d9b: stopping tablet replica
I20260619 18:50:14.578303 18956 tablet_replica.cc:330] T 485ff9db72c14d9ca26b2a9b2e921d50 P 8d3f261588f14e158cc4b8a3388f5136: stopping tablet replica
I20260619 18:50:14.578910 18956 raft_consensus.cc:2229] T 485ff9db72c14d9ca26b2a9b2e921d50 P 8d3f261588f14e158cc4b8a3388f5136 [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:50:14.579460 18954 raft_consensus.cc:2229] T 485ff9db72c14d9ca26b2a9b2e921d50 P a87fb8ce1b304e8e9dbc3e969e690d9b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:50:14.580052 18954 raft_consensus.cc:2258] T 485ff9db72c14d9ca26b2a9b2e921d50 P a87fb8ce1b304e8e9dbc3e969e690d9b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:50:14.580773 18956 raft_consensus.cc:2258] T 485ff9db72c14d9ca26b2a9b2e921d50 P 8d3f261588f14e158cc4b8a3388f5136 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:50:14.583343 18954 ts_tablet_manager.cc:1905] T 485ff9db72c14d9ca26b2a9b2e921d50 P a87fb8ce1b304e8e9dbc3e969e690d9b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:50:14.585361 18956 ts_tablet_manager.cc:1905] T 485ff9db72c14d9ca26b2a9b2e921d50 P 8d3f261588f14e158cc4b8a3388f5136: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:50:14.585937 18955 tablet_replica.cc:330] T 485ff9db72c14d9ca26b2a9b2e921d50 P fa3486d01e9045dcba230304040c37f1: stopping tablet replica
I20260619 18:50:14.587064 18955 raft_consensus.cc:2229] T 485ff9db72c14d9ca26b2a9b2e921d50 P fa3486d01e9045dcba230304040c37f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:50:14.588439 18926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:14.570008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36850 (local address 127.3.243.65:46855)
0619 18:50:14.570563 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0619 18:50:14.570569 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:50:14.570585 (+ 16us) server_negotiation.cc:407] Connection header received
0619 18:50:14.570636 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:50:14.570641 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:50:14.570731 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:50:14.570829 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:50:14.572707 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:14.574103 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:14.576032 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:14.576711 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:14.576831 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:14.580408 (+ 3577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:50:14.580422 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:50:14.580438 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:50:14.580484 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:50:14.584319 (+ 3835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:14.585229 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:14.585238 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:14.585242 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:14.585403 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:14.585856 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:14.585860 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:14.585864 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:14.586278 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:50:14.586605 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0619 18:50:14.587790 (+ 1185us) server_negotiation.cc:299] Negotiation successful
0619 18:50:14.588022 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434}
I20260619 18:50:14.588538 18955 raft_consensus.cc:2258] T 485ff9db72c14d9ca26b2a9b2e921d50 P fa3486d01e9045dcba230304040c37f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:50:14.589259 18647 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:50:14.590502 18955 ts_tablet_manager.cc:1905] T 485ff9db72c14d9ca26b2a9b2e921d50 P fa3486d01e9045dcba230304040c37f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:50:14.593053 18954 ts_tablet_manager.cc:1918] T 485ff9db72c14d9ca26b2a9b2e921d50 P a87fb8ce1b304e8e9dbc3e969e690d9b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:50:14.593196 18954 log.cc:1196] T 485ff9db72c14d9ca26b2a9b2e921d50 P a87fb8ce1b304e8e9dbc3e969e690d9b: Deleting WAL directory at /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-2/wal/wals/485ff9db72c14d9ca26b2a9b2e921d50
I20260619 18:50:14.593649 18954 ts_tablet_manager.cc:1939] T 485ff9db72c14d9ca26b2a9b2e921d50 P a87fb8ce1b304e8e9dbc3e969e690d9b: Deleting consensus metadata
I20260619 18:50:14.595403 18452 catalog_manager.cc:4841] TS a87fb8ce1b304e8e9dbc3e969e690d9b (127.3.243.67:34767): tablet 485ff9db72c14d9ca26b2a9b2e921d50 (table test-table [id=4a76f690ffe14db98311da09e793ba0f]) successfully deleted
I20260619 18:50:14.596588 18955 ts_tablet_manager.cc:1918] T 485ff9db72c14d9ca26b2a9b2e921d50 P fa3486d01e9045dcba230304040c37f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:50:14.596707 18955 log.cc:1196] T 485ff9db72c14d9ca26b2a9b2e921d50 P fa3486d01e9045dcba230304040c37f1: Deleting WAL directory at /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-1/wal/wals/485ff9db72c14d9ca26b2a9b2e921d50
I20260619 18:50:14.597103 18955 ts_tablet_manager.cc:1939] T 485ff9db72c14d9ca26b2a9b2e921d50 P fa3486d01e9045dcba230304040c37f1: Deleting consensus metadata
I20260619 18:50:14.598330 18452 catalog_manager.cc:4841] TS fa3486d01e9045dcba230304040c37f1 (127.3.243.66:46229): tablet 485ff9db72c14d9ca26b2a9b2e921d50 (table test-table [id=4a76f690ffe14db98311da09e793ba0f]) successfully deleted
I20260619 18:50:14.687186 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 18504
W20260619 18:50:14.703019 18453 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.243.65:46855 (error 108)
I20260619 18:50:14.703245 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 18656
I20260619 18:50:14.704844 18959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:14.704126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.65:46855 (local address 127.0.0.1:36862)
0619 18:50:14.704597 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:14.704691 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.65:46855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":245,"threads_started":1}
W20260619 18:50:14.705042 18453 catalog_manager.cc:4584] TS 8d3f261588f14e158cc4b8a3388f5136 (127.3.243.65:46855): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 485ff9db72c14d9ca26b2a9b2e921d50: Network error: Client connection negotiation failed: client connection to 127.3.243.65:46855: connect: Connection refused (error 111)
I20260619 18:50:14.716059 18928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:14.704041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33424 (local address 127.3.243.66:46229)
0619 18:50:14.704241 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 18:50:14.704247 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:50:14.704269 (+ 22us) server_negotiation.cc:407] Connection header received
0619 18:50:14.704444 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:50:14.704450 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:50:14.704525 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:50:14.704613 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:50:14.706031 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:14.706840 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:14.708054 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:14.708536 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:14.708644 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:14.711094 (+ 2450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:50:14.711105 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:50:14.711118 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:50:14.711149 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:50:14.713757 (+ 2608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:14.714375 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:14.714382 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:14.714385 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:14.714496 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:14.714798 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:14.714801 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:14.714803 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:14.715151 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:50:14.715357 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0619 18:50:14.715577 (+ 220us) server_negotiation.cc:299] Negotiation successful
0619 18:50:14.715725 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 18:50:14.716655 18779 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:50:14.758534 18959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:14.758056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.65:46855 (local address 127.0.0.1:36864)
0619 18:50:14.758263 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:14.758366 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.65:46855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260619 18:50:14.759555 18959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:14.759143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.65:46855 (local address 127.0.0.1:36868)
0619 18:50:14.759295 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:14.759377 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.65:46855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260619 18:50:14.816735 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 18656
I20260619 18:50:14.819835 18959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:14.819332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.65:46855 (local address 127.0.0.1:36878)
0619 18:50:14.819553 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:14.819659 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.65:46855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260619 18:50:14.820848 18959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:14.820466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.65:46855 (local address 127.0.0.1:36888)
0619 18:50:14.820613 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:14.820694 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.65:46855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260619 18:50:14.831358 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 18787
I20260619 18:50:14.843099 18930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:14.832054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33680 (local address 127.3.243.67:34767)
0619 18:50:14.832235 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 18:50:14.832239 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:50:14.832255 (+ 16us) server_negotiation.cc:407] Connection header received
0619 18:50:14.832386 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:50:14.832390 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:50:14.832454 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:50:14.832533 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:50:14.833763 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:14.834467 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:14.835656 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:14.836031 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:14.836114 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:14.838527 (+ 2413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:50:14.838539 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:50:14.838555 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:50:14.838582 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:50:14.840938 (+ 2356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:14.841549 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:14.841555 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:14.841557 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:14.841673 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:14.841948 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:14.841950 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:14.841952 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:14.842291 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:50:14.842497 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0619 18:50:14.842696 (+ 199us) server_negotiation.cc:299] Negotiation successful
0619 18:50:14.842851 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 18:50:14.843711 18910 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:50:14.901381 18959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:14.900946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.65:46855 (local address 127.0.0.1:36904)
0619 18:50:14.901132 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:14.901233 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.65:46855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260619 18:50:14.902403 18959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:14.902006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.65:46855 (local address 127.0.0.1:36906)
0619 18:50:14.902186 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:14.902256 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.65:46855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260619 18:50:14.929155 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 18787
I20260619 18:50:14.942096 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 18434
I20260619 18:50:14.943130 18497 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:50:15.051646 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 18434
I20260619 18:50:15.070168 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.126:39697
--webserver_interface=127.3.243.126
--webserver_port=39799
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab
--principal=kudu/127.3.243.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:46639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.126:39697
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:50:15.374248 18963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:50:15.374501 18963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:50:15.374626 18963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:50:15.381459 18963 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:50:15.381548 18963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:50:15.381644 18963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:50:15.381717 18963 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:50:15.381775 18963 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:50:15.390861 18963 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:46639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.243.126:39697
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab
--principal=kudu/127.3.243.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.126:39697
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.243.126
--webserver_port=39799
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:50:15.391979 18963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:50:15.393198 18963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:50:15.403934 18969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:50:15.403985 18971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:50:15.404592 18968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:50:15.404613 18963 server_base.cc:1034] running on GCE node
I20260619 18:50:15.405361 18963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:50:15.406610 18963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:50:15.407791 18963 hybrid_clock.cc:648] HybridClock initialized: now 1781895015407738 us; error 47 us; skew 500 ppm
I20260619 18:50:15.408093 18963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:50:15 dist-test-slave-m9wg krb5kdc[17883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895015, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:50:15.410902 18963 init.cc:377] Logged in from keytab as kudu/127.3.243.126@KRBTEST.COM (short username kudu)
I20260619 18:50:15.412278 18963 webserver.cc:460] Webserver started at http://127.3.243.126:39799/ using document root <none> and password file <none>
I20260619 18:50:15.412900 18963 fs_manager.cc:359] Metadata directory not provided
I20260619 18:50:15.413012 18963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:50:15.417932 18963 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 18:50:15.420480 18978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:50:15.421648 18963 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 18:50:15.421792 18963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/master-0/wal
uuid: "22ae91d05d6a4613906a3966068df987"
format_stamp: "Formatted at 2026-06-19 18:50:09 on dist-test-slave-m9wg"
server_key: "d2bfefb082a12cc1c57cf924a95d3392"
server_key_iv: "e8a328116c7198cab94c0c4d14d33ba9"
server_key_version: "encryptionkey@0"
I20260619 18:50:15.422679 18963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:50:15.454094 18963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:50:15.458245 18963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:50:15.458487 18963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:50:15.475100 18963 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.126:39697
I20260619 18:50:15.475124 19029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.126:39697 every 8 connection(s)
I20260619 18:50:15.476517 18963 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
I20260619 18:50:15.478798 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 18963
I20260619 18:50:15.479552 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.65:46855
--local_ip_for_outbound_sockets=127.3.243.65
--tserver_master_addrs=127.3.243.126:39697
--webserver_port=34531
--webserver_interface=127.3.243.65
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.65.keytab
--principal=kudu/127.3.243.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:46639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 18:50:15.483039 19030 sys_catalog.cc:263] Verifying existing consensus state
I20260619 18:50:15.486210 19030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22ae91d05d6a4613906a3966068df987: Bootstrap starting.
I20260619 18:50:15.513641 19030 log.cc:824] T 00000000000000000000000000000000 P 22ae91d05d6a4613906a3966068df987: Log is configured to *not* fsync() on all Append() calls
I20260619 18:50:15.534463 19030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22ae91d05d6a4613906a3966068df987: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 18:50:15.535137 19030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22ae91d05d6a4613906a3966068df987: Bootstrap complete.
I20260619 18:50:15.548586 19030 raft_consensus.cc:348] T 00000000000000000000000000000000 P 22ae91d05d6a4613906a3966068df987 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22ae91d05d6a4613906a3966068df987" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 39697 } }
I20260619 18:50:15.549774 19030 raft_consensus.cc:729] T 00000000000000000000000000000000 P 22ae91d05d6a4613906a3966068df987 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22ae91d05d6a4613906a3966068df987, State: Initialized, Role: FOLLOWER
I20260619 18:50:15.550465 19030 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22ae91d05d6a4613906a3966068df987 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22ae91d05d6a4613906a3966068df987" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 39697 } }
I20260619 18:50:15.550740 19030 raft_consensus.cc:388] T 00000000000000000000000000000000 P 22ae91d05d6a4613906a3966068df987 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:50:15.550860 19030 raft_consensus.cc:482] T 00000000000000000000000000000000 P 22ae91d05d6a4613906a3966068df987 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:50:15.551036 19030 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 22ae91d05d6a4613906a3966068df987 [term 2 FOLLOWER]: Advancing to term 3
I20260619 18:50:15.556483 19030 raft_consensus.cc:504] T 00000000000000000000000000000000 P 22ae91d05d6a4613906a3966068df987 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22ae91d05d6a4613906a3966068df987" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 39697 } }
I20260619 18:50:15.557085 19030 leader_election.cc:302] T 00000000000000000000000000000000 P 22ae91d05d6a4613906a3966068df987 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22ae91d05d6a4613906a3966068df987; no voters:
I20260619 18:50:15.557814 19030 leader_election.cc:288] T 00000000000000000000000000000000 P 22ae91d05d6a4613906a3966068df987 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 18:50:15.557983 19034 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 22ae91d05d6a4613906a3966068df987 [term 3 FOLLOWER]: Leader election won for term 3
I20260619 18:50:15.559317 19034 raft_consensus.cc:686] T 00000000000000000000000000000000 P 22ae91d05d6a4613906a3966068df987 [term 3 LEADER]: Becoming Leader. State: Replica: 22ae91d05d6a4613906a3966068df987, State: Running, Role: LEADER
I20260619 18:50:15.559823 19034 consensus_queue.cc:237] T 00000000000000000000000000000000 P 22ae91d05d6a4613906a3966068df987 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22ae91d05d6a4613906a3966068df987" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 39697 } }
I20260619 18:50:15.560035 19030 sys_catalog.cc:564] T 00000000000000000000000000000000 P 22ae91d05d6a4613906a3966068df987 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:50:15.562264 19036 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22ae91d05d6a4613906a3966068df987 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 22ae91d05d6a4613906a3966068df987. Latest consensus state: current_term: 3 leader_uuid: "22ae91d05d6a4613906a3966068df987" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22ae91d05d6a4613906a3966068df987" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 39697 } } }
I20260619 18:50:15.562242 19035 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22ae91d05d6a4613906a3966068df987 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "22ae91d05d6a4613906a3966068df987" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22ae91d05d6a4613906a3966068df987" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 39697 } } }
I20260619 18:50:15.562662 19035 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22ae91d05d6a4613906a3966068df987 [sys.catalog]: This master's current role is: LEADER
I20260619 18:50:15.562662 19036 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22ae91d05d6a4613906a3966068df987 [sys.catalog]: This master's current role is: LEADER
I20260619 18:50:15.563422 19040 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:50:15.574972 19040 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:50:15.576493 19040 catalog_manager.cc:1259] Loaded cluster ID: 13dc07384b5e42a49cc7067630c5ac3e
I20260619 18:50:15.576654 19040 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260619 18:50:15.578729 19040 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f58059ee9ad google::LogMessage::Fail() at ??:0
@ 0x7f58059f188a google::LogMessage::SendToLog() at ??:0
@ 0x7f58059ee4de google::LogMessage::Flush() at ??:0
@ 0x7f58059efa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f581b421512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f581b41e976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f581b4a6610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f581b4a6439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f580863a345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f580863d91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f580863d799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f5808618eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f580b8f26db start_thread at ??:0
@ 0x7f580488861f clone at ??:0
W20260619 18:50:15.890831 19032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:50:15.891095 19032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:50:15.891185 19032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:50:15.898182 19032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:50:15.898283 19032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:50:15.898398 19032 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.65
I20260619 18:50:15.907730 19032 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:46639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.65.keytab
--principal=kudu/127.3.243.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.65:46855
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.243.65
--webserver_port=34531
--tserver_master_addrs=127.3.243.126:39697
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.65
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:50:15.908947 19032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:50:15.910166 19032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:50:15.921939 19069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:50:15.922044 19070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:50:15.922076 19072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:50:15.922727 19032 server_base.cc:1034] running on GCE node
I20260619 18:50:15.923267 19032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:50:15.924080 19032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:50:15.925299 19032 hybrid_clock.cc:648] HybridClock initialized: now 1781895015925246 us; error 43 us; skew 500 ppm
I20260619 18:50:15.925570 19032 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:50:15 dist-test-slave-m9wg krb5kdc[17883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895015, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:50:15.928330 19032 init.cc:377] Logged in from keytab as kudu/127.3.243.65@KRBTEST.COM (short username kudu)
I20260619 18:50:15.929627 19032 webserver.cc:460] Webserver started at http://127.3.243.65:34531/ using document root <none> and password file <none>
I20260619 18:50:15.930256 19032 fs_manager.cc:359] Metadata directory not provided
I20260619 18:50:15.930372 19032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:50:15.935369 19032 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 18:50:15.938050 19079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:50:15.939260 19032 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 18:50:15.939410 19032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-0/wal
uuid: "8d3f261588f14e158cc4b8a3388f5136"
format_stamp: "Formatted at 2026-06-19 18:50:10 on dist-test-slave-m9wg"
server_key: "4512017085bc64da101ab267e0829c0a"
server_key_iv: "f9aa9a356b97e4913808d79ae42c36f9"
server_key_version: "encryptionkey@0"
I20260619 18:50:15.940375 19032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:50:15.969380 19032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:50:15.973044 19032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:50:15.973285 19032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:50:15.974200 19032 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:50:15.975914 19086 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260619 18:50:15.981315 19032 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 18:50:15.981426 19032 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260619 18:50:15.981544 19032 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 18:50:15.984547 19032 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 18:50:15.984627 19032 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.003s sys 0.000s
I20260619 18:50:15.984958 19086 tablet_bootstrap.cc:492] T 485ff9db72c14d9ca26b2a9b2e921d50 P 8d3f261588f14e158cc4b8a3388f5136: Bootstrap starting.
I20260619 18:50:15.992455 19086 log.cc:824] T 485ff9db72c14d9ca26b2a9b2e921d50 P 8d3f261588f14e158cc4b8a3388f5136: Log is configured to *not* fsync() on all Append() calls
I20260619 18:50:16.008885 19086 tablet_bootstrap.cc:492] T 485ff9db72c14d9ca26b2a9b2e921d50 P 8d3f261588f14e158cc4b8a3388f5136: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 18:50:16.009630 19086 tablet_bootstrap.cc:492] T 485ff9db72c14d9ca26b2a9b2e921d50 P 8d3f261588f14e158cc4b8a3388f5136: Bootstrap complete.
I20260619 18:50:16.010192 19086 ts_tablet_manager.cc:1397] T 485ff9db72c14d9ca26b2a9b2e921d50 P 8d3f261588f14e158cc4b8a3388f5136: Time spent bootstrapping tablet: real 0.026s user 0.019s sys 0.004s
I20260619 18:50:16.019922 19032 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.65:46855
I20260619 18:50:16.019941 19193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.65:46855 every 8 connection(s)
I20260619 18:50:16.021411 19032 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-0/data/info.pb
I20260619 18:50:16.021118 19086 raft_consensus.cc:348] T 485ff9db72c14d9ca26b2a9b2e921d50 P 8d3f261588f14e158cc4b8a3388f5136 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d3f261588f14e158cc4b8a3388f5136" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 46855 } } peers { permanent_uuid: "fa3486d01e9045dcba230304040c37f1" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 46229 } } peers { permanent_uuid: "a87fb8ce1b304e8e9dbc3e969e690d9b" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 34767 } }
I20260619 18:50:16.022785 19086 raft_consensus.cc:729] T 485ff9db72c14d9ca26b2a9b2e921d50 P 8d3f261588f14e158cc4b8a3388f5136 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d3f261588f14e158cc4b8a3388f5136, State: Initialized, Role: FOLLOWER
I20260619 18:50:16.023428 19086 consensus_queue.cc:260] T 485ff9db72c14d9ca26b2a9b2e921d50 P 8d3f261588f14e158cc4b8a3388f5136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d3f261588f14e158cc4b8a3388f5136" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 46855 } } peers { permanent_uuid: "fa3486d01e9045dcba230304040c37f1" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 46229 } } peers { permanent_uuid: "a87fb8ce1b304e8e9dbc3e969e690d9b" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 34767 } }
I20260619 18:50:16.024940 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 19032
I20260619 18:50:16.024950 19086 ts_tablet_manager.cc:1428] T 485ff9db72c14d9ca26b2a9b2e921d50 P 8d3f261588f14e158cc4b8a3388f5136: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20260619 18:50:16.025565 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.66:46229
--local_ip_for_outbound_sockets=127.3.243.66
--tserver_master_addrs=127.3.243.126:39697
--webserver_port=45245
--webserver_interface=127.3.243.66
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.66.keytab
--principal=kudu/127.3.243.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:46639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 18:50:16.026051 19197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:16.024017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:39697 (local address 127.3.243.65:52765)
0619 18:50:16.024964 (+ 947us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:16.025269 (+ 305us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:39697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":626,"thread_start_us":378,"threads_started":1}
W20260619 18:50:16.026660 19194 heartbeater.cc:644] Failed to heartbeat to 127.3.243.126:39697 (0 consecutive failures): Network error: Failed to ping master at 127.3.243.126:39697: Client connection negotiation failed: client connection to 127.3.243.126:39697: connect: Connection refused (error 111)
I20260619 18:50:16.027392 19197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:16.027103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:39697 (local address 127.3.243.65:49197)
0619 18:50:16.027236 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:16.027300 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:39697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260619 18:50:16.028192 19197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:16.027860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:39697 (local address 127.3.243.65:33083)
0619 18:50:16.027985 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:16.028051 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:39697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260619 18:50:16.028478 19194 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 18:50:16.329452 19198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:50:16.329692 19198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:50:16.329779 19198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:50:16.336642 19198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:50:16.336737 19198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:50:16.336849 19198 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.66
I20260619 18:50:16.346017 19198 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:46639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.66.keytab
--principal=kudu/127.3.243.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.66:46229
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.243.66
--webserver_port=45245
--tserver_master_addrs=127.3.243.126:39697
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.66
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:50:16.347153 19198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:50:16.348409 19198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:50:16.360575 19203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:50:16.360558 19206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:50:16.360523 19204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:50:16.361377 19198 server_base.cc:1034] running on GCE node
I20260619 18:50:16.361944 19198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:50:16.362793 19198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:50:16.363982 19198 hybrid_clock.cc:648] HybridClock initialized: now 1781895016363931 us; error 45 us; skew 500 ppm
I20260619 18:50:16.364320 19198 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:50:16 dist-test-slave-m9wg krb5kdc[17883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895016, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:50:16.367197 19198 init.cc:377] Logged in from keytab as kudu/127.3.243.66@KRBTEST.COM (short username kudu)
I20260619 18:50:16.368461 19198 webserver.cc:460] Webserver started at http://127.3.243.66:45245/ using document root <none> and password file <none>
I20260619 18:50:16.369046 19198 fs_manager.cc:359] Metadata directory not provided
I20260619 18:50:16.369136 19198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:50:16.373901 19198 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 18:50:16.376385 19213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:50:16.377614 19198 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 18:50:16.377751 19198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-1/wal
uuid: "fa3486d01e9045dcba230304040c37f1"
format_stamp: "Formatted at 2026-06-19 18:50:10 on dist-test-slave-m9wg"
server_key: "6c3293924cc918b396555e43d5f1d38d"
server_key_iv: "9838ff3f14c6f992483f7f08be0ff4a7"
server_key_version: "encryptionkey@0"
I20260619 18:50:16.378636 19198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:50:16.390118 19198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:50:16.393965 19198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:50:16.394187 19198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:50:16.395217 19198 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:50:16.396505 19198 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:50:16.396588 19198 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:50:16.396683 19198 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:50:16.396754 19198 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:50:16.425253 19198 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.66:46229
I20260619 18:50:16.425308 19325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.66:46229 every 8 connection(s)
I20260619 18:50:16.426608 19198 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-1/data/info.pb
I20260619 18:50:16.431224 19328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:16.429436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:39697 (local address 127.3.243.66:42271)
0619 18:50:16.430135 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:16.430379 (+ 244us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:39697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":268,"threads_started":1}
W20260619 18:50:16.431928 19326 heartbeater.cc:644] Failed to heartbeat to 127.3.243.126:39697 (0 consecutive failures): Network error: Failed to ping master at 127.3.243.126:39697: Client connection negotiation failed: client connection to 127.3.243.126:39697: connect: Connection refused (error 111)
I20260619 18:50:16.432787 19328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:16.432446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:39697 (local address 127.3.243.66:36951)
0619 18:50:16.432611 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:16.432678 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:39697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260619 18:50:16.433604 19328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:16.433275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:39697 (local address 127.3.243.66:47525)
0619 18:50:16.433421 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:16.433494 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:39697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260619 18:50:16.433815 19326 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 18:50:16.435410 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 19198
I20260619 18:50:16.435966 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.67:34767
--local_ip_for_outbound_sockets=127.3.243.67
--tserver_master_addrs=127.3.243.126:39697
--webserver_port=38533
--webserver_interface=127.3.243.67
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.67.keytab
--principal=kudu/127.3.243.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:46639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:50:16.741536 19329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:50:16.741798 19329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:50:16.741922 19329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:50:16.748813 19329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:50:16.748910 19329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:50:16.749051 19329 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.67
I20260619 18:50:16.758381 19329 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:46639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.67.keytab
--principal=kudu/127.3.243.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.67:34767
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.243.67
--webserver_port=38533
--tserver_master_addrs=127.3.243.126:39697
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.67
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:50:16.759763 19329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:50:16.761108 19329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:50:16.773000 19335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:50:16.772971 19337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:50:16.773182 19334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:50:16.773619 19329 server_base.cc:1034] running on GCE node
I20260619 18:50:16.774180 19329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:50:16.775095 19329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:50:16.776242 19329 hybrid_clock.cc:648] HybridClock initialized: now 1781895016776184 us; error 47 us; skew 500 ppm
I20260619 18:50:16.776530 19329 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:50:16 dist-test-slave-m9wg krb5kdc[17883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895016, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:50:16.779405 19329 init.cc:377] Logged in from keytab as kudu/127.3.243.67@KRBTEST.COM (short username kudu)
I20260619 18:50:16.780637 19329 webserver.cc:460] Webserver started at http://127.3.243.67:38533/ using document root <none> and password file <none>
I20260619 18:50:16.781250 19329 fs_manager.cc:359] Metadata directory not provided
I20260619 18:50:16.781338 19329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:50:16.786252 19329 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 18:50:16.788668 19344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:50:16.789923 19329 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:50:16.790071 19329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-2/wal
uuid: "a87fb8ce1b304e8e9dbc3e969e690d9b"
format_stamp: "Formatted at 2026-06-19 18:50:11 on dist-test-slave-m9wg"
server_key: "38f5df90f25296131d3953cc271cb82b"
server_key_iv: "e66a5563157d00381fb091c0243bea8d"
server_key_version: "encryptionkey@0"
I20260619 18:50:16.791044 19329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:50:16.811573 19329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:50:16.815361 19329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:50:16.815584 19329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:50:16.816426 19329 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:50:16.817631 19329 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:50:16.817727 19329 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:50:16.817835 19329 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:50:16.817906 19329 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:50:16.846380 19329 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.67:34767
I20260619 18:50:16.846431 19456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.67:34767 every 8 connection(s)
I20260619 18:50:16.847750 19329 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894810715282-4045-0/minicluster-data/ts-2/data/info.pb
I20260619 18:50:16.852351 19459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:16.850512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:39697 (local address 127.3.243.67:45327)
0619 18:50:16.851184 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:16.851420 (+ 236us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:39697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":244,"threads_started":1}
W20260619 18:50:16.852998 19457 heartbeater.cc:644] Failed to heartbeat to 127.3.243.126:39697 (0 consecutive failures): Network error: Failed to ping master at 127.3.243.126:39697: Client connection negotiation failed: client connection to 127.3.243.126:39697: connect: Connection refused (error 111)
I20260619 18:50:16.853689 19459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:16.853406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:39697 (local address 127.3.243.67:53469)
0619 18:50:16.853532 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:16.853595 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:39697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260619 18:50:16.854408 19459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:16.854112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:39697 (local address 127.3.243.67:59533)
0619 18:50:16.854237 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:16.854308 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:39697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260619 18:50:16.854614 19457 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 18:50:16.855818 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 19329
I20260619 18:50:17.029743 19461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:17.028949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:39697 (local address 127.3.243.65:58737)
0619 18:50:17.029419 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:17.029540 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:39697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":201,"threads_started":1}
I20260619 18:50:17.435014 19462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:17.434270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:39697 (local address 127.3.243.66:36267)
0619 18:50:17.434698 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:17.434793 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:39697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":218,"threads_started":1}
I20260619 18:50:17.475708 19463 raft_consensus.cc:482] T 485ff9db72c14d9ca26b2a9b2e921d50 P 8d3f261588f14e158cc4b8a3388f5136 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:50:17.475978 19463 raft_consensus.cc:504] T 485ff9db72c14d9ca26b2a9b2e921d50 P 8d3f261588f14e158cc4b8a3388f5136 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d3f261588f14e158cc4b8a3388f5136" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 46855 } } peers { permanent_uuid: "fa3486d01e9045dcba230304040c37f1" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 46229 } } peers { permanent_uuid: "a87fb8ce1b304e8e9dbc3e969e690d9b" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 34767 } }
I20260619 18:50:17.477442 19463 leader_election.cc:288] T 485ff9db72c14d9ca26b2a9b2e921d50 P 8d3f261588f14e158cc4b8a3388f5136 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fa3486d01e9045dcba230304040c37f1 (127.3.243.66:46229), a87fb8ce1b304e8e9dbc3e969e690d9b (127.3.243.67:34767)
Jun 19 18:50:17 dist-test-slave-m9wg krb5kdc[17883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895015, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.65@KRBTEST.COM for kudu/127.3.243.67@KRBTEST.COM
Jun 19 18:50:17 dist-test-slave-m9wg krb5kdc[17883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895015, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.65@KRBTEST.COM for kudu/127.3.243.66@KRBTEST.COM
I20260619 18:50:17.497157 19464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:17.477973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.65:42019 (local address 127.3.243.66:46229)
0619 18:50:17.479562 (+ 1589us) server_negotiation.cc:206] Beginning negotiation
0619 18:50:17.479574 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 18:50:17.479611 (+ 37us) server_negotiation.cc:407] Connection header received
0619 18:50:17.481117 (+ 1506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:50:17.481126 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:50:17.481529 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:50:17.481835 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:50:17.483794 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:17.484969 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:17.486565 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:17.487149 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:17.487298 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:17.491031 (+ 3733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:50:17.491037 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:50:17.491048 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:50:17.491080 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:50:17.493898 (+ 2818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:17.494576 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:17.494582 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:17.494587 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:17.494763 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:17.495133 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:17.495138 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:17.495141 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:17.495664 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:50:17.495816 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 18:50:17.496433 (+ 617us) server_negotiation.cc:299] Negotiation successful
0619 18:50:17.496789 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1433,"thread_start_us":1282,"threads_started":1}
I20260619 18:50:17.497155 19461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:17.477746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.66:46229 (local address 127.3.243.65:42019)
0619 18:50:17.477900 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:17.477932 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0619 18:50:17.478470 (+ 538us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:50:17.481953 (+ 3483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:50:17.481963 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:50:17.482836 (+ 873us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:50:17.483624 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:50:17.483643 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:50:17.485139 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:50:17.485142 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:50:17.486433 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:50:17.486445 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:50:17.487377 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:50:17.487384 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:50:17.487585 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:17.488249 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:50:17.488274 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:50:17.490805 (+ 2531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:50:17.494021 (+ 3216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:50:17.494026 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:50:17.494030 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:50:17.494462 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:50:17.494884 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:50:17.494889 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:50:17.494893 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:50:17.495044 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:50:17.495789 (+ 745us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:50:17.495794 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:50:17.495954 (+ 160us) client_negotiation.cc:769] Sending connection context
0619 18:50:17.496213 (+ 259us) client_negotiation.cc:240] Negotiation successful
0619 18:50:17.496722 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":119,"spinlock_wait_cycles":16768}
I20260619 18:50:17.497156 19466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:17.478477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.65:34301 (local address 127.3.243.67:34767)
0619 18:50:17.479869 (+ 1392us) server_negotiation.cc:206] Beginning negotiation
0619 18:50:17.479882 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 18:50:17.479911 (+ 29us) server_negotiation.cc:407] Connection header received
0619 18:50:17.481289 (+ 1378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:50:17.481297 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:50:17.481657 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:50:17.481929 (+ 272us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:50:17.483799 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:17.484964 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:17.486558 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:17.487145 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:17.487310 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:17.490813 (+ 3503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:50:17.490821 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:50:17.490833 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:50:17.490873 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:50:17.493515 (+ 2642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:17.494400 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:17.494407 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:17.494412 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:17.494560 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:17.495002 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:17.495007 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:17.495010 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:17.495528 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:50:17.495685 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 18:50:17.496430 (+ 745us) server_negotiation.cc:299] Negotiation successful
0619 18:50:17.496754 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1248,"thread_start_us":1142,"threads_started":1}
I20260619 18:50:17.497252 19465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:17.478183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.67:34767 (local address 127.3.243.65:34301)
0619 18:50:17.479297 (+ 1114us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:17.479314 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 18:50:17.479432 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:50:17.481952 (+ 2520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:50:17.481962 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:50:17.482852 (+ 890us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:50:17.483623 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:50:17.483642 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:50:17.485135 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:50:17.485139 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:50:17.486437 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:50:17.486448 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:50:17.487333 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:50:17.487339 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:50:17.487586 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:17.488321 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:50:17.488337 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:50:17.490571 (+ 2234us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:50:17.493639 (+ 3068us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:50:17.493647 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:50:17.493661 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:50:17.494280 (+ 619us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:50:17.494676 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:50:17.494681 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:50:17.494685 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:50:17.494894 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:50:17.495659 (+ 765us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:50:17.495666 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:50:17.495951 (+ 285us) client_negotiation.cc:769] Sending connection context
0619 18:50:17.496213 (+ 262us) client_negotiation.cc:240] Negotiation successful
0619 18:50:17.496861 (+ 648us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":981,"mutex_wait_us":324,"spinlock_wait_cycles":11008,"thread_start_us":806,"threads_started":1}
W20260619 18:50:17.500530 19081 leader_election.cc:341] T 485ff9db72c14d9ca26b2a9b2e921d50 P 8d3f261588f14e158cc4b8a3388f5136 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a87fb8ce1b304e8e9dbc3e969e690d9b (127.3.243.67:34767): Not found: Tablet not found: 485ff9db72c14d9ca26b2a9b2e921d50
W20260619 18:50:17.501405 19081 leader_election.cc:341] T 485ff9db72c14d9ca26b2a9b2e921d50 P 8d3f261588f14e158cc4b8a3388f5136 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fa3486d01e9045dcba230304040c37f1 (127.3.243.66:46229): Not found: Tablet not found: 485ff9db72c14d9ca26b2a9b2e921d50
I20260619 18:50:17.501542 19081 leader_election.cc:302] T 485ff9db72c14d9ca26b2a9b2e921d50 P 8d3f261588f14e158cc4b8a3388f5136 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8d3f261588f14e158cc4b8a3388f5136; no voters: a87fb8ce1b304e8e9dbc3e969e690d9b, fa3486d01e9045dcba230304040c37f1
I20260619 18:50:17.501936 19463 raft_consensus.cc:2726] T 485ff9db72c14d9ca26b2a9b2e921d50 P 8d3f261588f14e158cc4b8a3388f5136 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 18:50:17.855897 19467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:17.855104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:39697 (local address 127.3.243.67:38071)
0619 18:50:17.855542 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:17.855676 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:39697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":213,"threads_started":1}
I20260619 18:50:18.031144 19468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:18.030518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:39697 (local address 127.3.243.65:55289)
0619 18:50:18.030900 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:18.030994 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:39697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":189,"threads_started":1}
I20260619 18:50:18.436465 19469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:18.435810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:39697 (local address 127.3.243.66:45489)
0619 18:50:18.436236 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:18.436315 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:39697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":213,"threads_started":1}
I20260619 18:50:18.857339 19470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:18.856708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:39697 (local address 127.3.243.67:44745)
0619 18:50:18.857108 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:18.857190 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:39697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":208,"threads_started":1}
I20260619 18:50:19.012270 19471 raft_consensus.cc:482] T 485ff9db72c14d9ca26b2a9b2e921d50 P 8d3f261588f14e158cc4b8a3388f5136 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:50:19.012462 19471 raft_consensus.cc:504] T 485ff9db72c14d9ca26b2a9b2e921d50 P 8d3f261588f14e158cc4b8a3388f5136 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d3f261588f14e158cc4b8a3388f5136" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 46855 } } peers { permanent_uuid: "fa3486d01e9045dcba230304040c37f1" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 46229 } } peers { permanent_uuid: "a87fb8ce1b304e8e9dbc3e969e690d9b" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 34767 } }
I20260619 18:50:19.012990 19471 leader_election.cc:288] T 485ff9db72c14d9ca26b2a9b2e921d50 P 8d3f261588f14e158cc4b8a3388f5136 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fa3486d01e9045dcba230304040c37f1 (127.3.243.66:46229), a87fb8ce1b304e8e9dbc3e969e690d9b (127.3.243.67:34767)
W20260619 18:50:19.013860 19081 leader_election.cc:341] T 485ff9db72c14d9ca26b2a9b2e921d50 P 8d3f261588f14e158cc4b8a3388f5136 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fa3486d01e9045dcba230304040c37f1 (127.3.243.66:46229): Not found: Tablet not found: 485ff9db72c14d9ca26b2a9b2e921d50
W20260619 18:50:19.014058 19081 leader_election.cc:341] T 485ff9db72c14d9ca26b2a9b2e921d50 P 8d3f261588f14e158cc4b8a3388f5136 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a87fb8ce1b304e8e9dbc3e969e690d9b (127.3.243.67:34767): Not found: Tablet not found: 485ff9db72c14d9ca26b2a9b2e921d50
I20260619 18:50:19.014168 19081 leader_election.cc:302] T 485ff9db72c14d9ca26b2a9b2e921d50 P 8d3f261588f14e158cc4b8a3388f5136 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8d3f261588f14e158cc4b8a3388f5136; no voters: a87fb8ce1b304e8e9dbc3e969e690d9b, fa3486d01e9045dcba230304040c37f1
I20260619 18:50:19.014377 19471 raft_consensus.cc:2726] T 485ff9db72c14d9ca26b2a9b2e921d50 P 8d3f261588f14e158cc4b8a3388f5136 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 18:50:19.032641 19472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:19.031907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:39697 (local address 127.3.243.65:56611)
0619 18:50:19.032365 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:19.032465 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:39697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":234,"threads_started":1}
I20260619 18:50:19.437886 19473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:19.437232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:39697 (local address 127.3.243.66:40755)
0619 18:50:19.437657 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:19.437730 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:39697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":238,"threads_started":1}
I20260619 18:50:19.858665 19474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:19.858050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:39697 (local address 127.3.243.67:48661)
0619 18:50:19.858429 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:19.858519 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:39697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":193,"threads_started":1}
I20260619 18:50:20.033970 19475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:20.033359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:39697 (local address 127.3.243.65:40167)
0619 18:50:20.033752 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:20.033826 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:39697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":213,"threads_started":1}
I20260619 18:50:20.439378 19476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:20.438687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:39697 (local address 127.3.243.66:45167)
0619 18:50:20.439129 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:20.439205 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:39697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":218,"threads_started":1}
I20260619 18:50:20.581833 19477 raft_consensus.cc:482] T 485ff9db72c14d9ca26b2a9b2e921d50 P 8d3f261588f14e158cc4b8a3388f5136 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:50:20.582011 19477 raft_consensus.cc:504] T 485ff9db72c14d9ca26b2a9b2e921d50 P 8d3f261588f14e158cc4b8a3388f5136 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d3f261588f14e158cc4b8a3388f5136" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 46855 } } peers { permanent_uuid: "fa3486d01e9045dcba230304040c37f1" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 46229 } } peers { permanent_uuid: "a87fb8ce1b304e8e9dbc3e969e690d9b" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 34767 } }
I20260619 18:50:20.582617 19477 leader_election.cc:288] T 485ff9db72c14d9ca26b2a9b2e921d50 P 8d3f261588f14e158cc4b8a3388f5136 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fa3486d01e9045dcba230304040c37f1 (127.3.243.66:46229), a87fb8ce1b304e8e9dbc3e969e690d9b (127.3.243.67:34767)
W20260619 18:50:20.583539 19081 leader_election.cc:341] T 485ff9db72c14d9ca26b2a9b2e921d50 P 8d3f261588f14e158cc4b8a3388f5136 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fa3486d01e9045dcba230304040c37f1 (127.3.243.66:46229): Not found: Tablet not found: 485ff9db72c14d9ca26b2a9b2e921d50
W20260619 18:50:20.583885 19081 leader_election.cc:341] T 485ff9db72c14d9ca26b2a9b2e921d50 P 8d3f261588f14e158cc4b8a3388f5136 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a87fb8ce1b304e8e9dbc3e969e690d9b (127.3.243.67:34767): Not found: Tablet not found: 485ff9db72c14d9ca26b2a9b2e921d50
I20260619 18:50:20.584030 19081 leader_election.cc:302] T 485ff9db72c14d9ca26b2a9b2e921d50 P 8d3f261588f14e158cc4b8a3388f5136 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8d3f261588f14e158cc4b8a3388f5136; no voters: a87fb8ce1b304e8e9dbc3e969e690d9b, fa3486d01e9045dcba230304040c37f1
I20260619 18:50:20.584288 19477 raft_consensus.cc:2726] T 485ff9db72c14d9ca26b2a9b2e921d50 P 8d3f261588f14e158cc4b8a3388f5136 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 18:50:20.860062 19478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:20.859408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:39697 (local address 127.3.243.67:46167)
0619 18:50:20.859808 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:20.859881 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:39697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":203,"threads_started":1}
I20260619 18:50:21.035610 19479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:21.034886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:39697 (local address 127.3.243.65:48911)
0619 18:50:21.035338 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:21.035420 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:39697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":210,"threads_started":1}
I20260619 18:50:21.440877 19480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:21.440234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:39697 (local address 127.3.243.66:53209)
0619 18:50:21.440638 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:21.440715 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:39697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":214,"threads_started":1}
I20260619 18:50:21.861583 19481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:21.860868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:39697 (local address 127.3.243.67:48869)
0619 18:50:21.861339 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:21.861420 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:39697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":262,"threads_started":1}
I20260619 18:50:22.036993 19482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:22.036370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:39697 (local address 127.3.243.65:46273)
0619 18:50:22.036753 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:22.036847 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:39697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":197,"threads_started":1}
I20260619 18:50:22.412575 19483 raft_consensus.cc:482] T 485ff9db72c14d9ca26b2a9b2e921d50 P 8d3f261588f14e158cc4b8a3388f5136 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:50:22.412770 19483 raft_consensus.cc:504] T 485ff9db72c14d9ca26b2a9b2e921d50 P 8d3f261588f14e158cc4b8a3388f5136 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d3f261588f14e158cc4b8a3388f5136" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 46855 } } peers { permanent_uuid: "fa3486d01e9045dcba230304040c37f1" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 46229 } } peers { permanent_uuid: "a87fb8ce1b304e8e9dbc3e969e690d9b" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 34767 } }
I20260619 18:50:22.413452 19483 leader_election.cc:288] T 485ff9db72c14d9ca26b2a9b2e921d50 P 8d3f261588f14e158cc4b8a3388f5136 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fa3486d01e9045dcba230304040c37f1 (127.3.243.66:46229), a87fb8ce1b304e8e9dbc3e969e690d9b (127.3.243.67:34767)
W20260619 18:50:22.414413 19081 leader_election.cc:341] T 485ff9db72c14d9ca26b2a9b2e921d50 P 8d3f261588f14e158cc4b8a3388f5136 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fa3486d01e9045dcba230304040c37f1 (127.3.243.66:46229): Not found: Tablet not found: 485ff9db72c14d9ca26b2a9b2e921d50
W20260619 18:50:22.414625 19081 leader_election.cc:341] T 485ff9db72c14d9ca26b2a9b2e921d50 P 8d3f261588f14e158cc4b8a3388f5136 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a87fb8ce1b304e8e9dbc3e969e690d9b (127.3.243.67:34767): Not found: Tablet not found: 485ff9db72c14d9ca26b2a9b2e921d50
I20260619 18:50:22.414712 19081 leader_election.cc:302] T 485ff9db72c14d9ca26b2a9b2e921d50 P 8d3f261588f14e158cc4b8a3388f5136 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8d3f261588f14e158cc4b8a3388f5136; no voters: a87fb8ce1b304e8e9dbc3e969e690d9b, fa3486d01e9045dcba230304040c37f1
I20260619 18:50:22.414978 19483 raft_consensus.cc:2726] T 485ff9db72c14d9ca26b2a9b2e921d50 P 8d3f261588f14e158cc4b8a3388f5136 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 18:50:22.442337 19484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:22.441664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:39697 (local address 127.3.243.66:32957)
0619 18:50:22.442085 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:22.442164 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:39697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":215,"threads_started":1}
I20260619 18:50:22.862998 19485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:22.862310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:39697 (local address 127.3.243.67:55141)
0619 18:50:22.862748 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:22.862832 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:39697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":212,"threads_started":1}
I20260619 18:50:23.038419 19486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:23.037706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:39697 (local address 127.3.243.65:36887)
0619 18:50:23.038134 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:23.038214 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:39697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":217,"threads_started":1}
I20260619 18:50:23.443991 19487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:23.443153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:39697 (local address 127.3.243.66:42817)
0619 18:50:23.443702 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:23.443788 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:39697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":238,"threads_started":1}
I20260619 18:50:23.864575 19488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:23.863792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:39697 (local address 127.3.243.67:52469)
0619 18:50:23.864313 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:23.864397 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:39697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":240,"threads_started":1}
I20260619 18:50:24.040040 19489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:24.039234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:39697 (local address 127.3.243.65:57293)
0619 18:50:24.039748 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:24.039881 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:39697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":245,"threads_started":1}
I20260619 18:50:24.445484 19490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:24.444808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:39697 (local address 127.3.243.66:60165)
0619 18:50:24.445222 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:24.445310 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:39697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":207,"threads_started":1}
I20260619 18:50:24.866024 19491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:24.865395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:39697 (local address 127.3.243.67:33645)
0619 18:50:24.865756 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:24.865834 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:39697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":179,"threads_started":1}
I20260619 18:50:25.041508 19492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:25.040851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:39697 (local address 127.3.243.65:45233)
0619 18:50:25.041255 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:25.041337 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:39697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":200,"threads_started":1}
I20260619 18:50:25.447003 19493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:25.446337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:39697 (local address 127.3.243.66:44949)
0619 18:50:25.446756 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:25.446835 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:39697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":201,"threads_started":1}
I20260619 18:50:25.867650 19494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:25.866876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:39697 (local address 127.3.243.67:53461)
0619 18:50:25.867382 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:25.867480 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:39697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":224,"threads_started":1}
I20260619 18:50:26.043169 19495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:26.042484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:39697 (local address 127.3.243.65:46033)
0619 18:50:26.042936 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:26.043013 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:39697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":217,"threads_started":1}
I20260619 18:50:26.448506 19496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:26.447808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:39697 (local address 127.3.243.66:55195)
0619 18:50:26.448252 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:26.448328 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:39697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":210,"threads_started":1}
I20260619 18:50:26.869238 19497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:26.868525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:39697 (local address 127.3.243.67:59935)
0619 18:50:26.868986 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:26.869066 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:39697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":257,"threads_started":1}
I20260619 18:50:27.044616 19498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:27.043845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:39697 (local address 127.3.243.65:56385)
0619 18:50:27.044332 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:27.044418 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:39697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":210,"threads_started":1}
I20260619 18:50:27.450008 19499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:27.449321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:39697 (local address 127.3.243.66:44849)
0619 18:50:27.449756 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:27.449831 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:39697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":210,"threads_started":1}
I20260619 18:50:27.476004 19500 raft_consensus.cc:482] T 485ff9db72c14d9ca26b2a9b2e921d50 P 8d3f261588f14e158cc4b8a3388f5136 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:50:27.476248 19500 raft_consensus.cc:504] T 485ff9db72c14d9ca26b2a9b2e921d50 P 8d3f261588f14e158cc4b8a3388f5136 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d3f261588f14e158cc4b8a3388f5136" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 46855 } } peers { permanent_uuid: "fa3486d01e9045dcba230304040c37f1" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 46229 } } peers { permanent_uuid: "a87fb8ce1b304e8e9dbc3e969e690d9b" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 34767 } }
I20260619 18:50:27.476845 19500 leader_election.cc:288] T 485ff9db72c14d9ca26b2a9b2e921d50 P 8d3f261588f14e158cc4b8a3388f5136 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fa3486d01e9045dcba230304040c37f1 (127.3.243.66:46229), a87fb8ce1b304e8e9dbc3e969e690d9b (127.3.243.67:34767)
W20260619 18:50:27.477742 19081 leader_election.cc:341] T 485ff9db72c14d9ca26b2a9b2e921d50 P 8d3f261588f14e158cc4b8a3388f5136 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fa3486d01e9045dcba230304040c37f1 (127.3.243.66:46229): Not found: Tablet not found: 485ff9db72c14d9ca26b2a9b2e921d50
W20260619 18:50:27.477967 19081 leader_election.cc:341] T 485ff9db72c14d9ca26b2a9b2e921d50 P 8d3f261588f14e158cc4b8a3388f5136 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a87fb8ce1b304e8e9dbc3e969e690d9b (127.3.243.67:34767): Not found: Tablet not found: 485ff9db72c14d9ca26b2a9b2e921d50
I20260619 18:50:27.478075 19081 leader_election.cc:302] T 485ff9db72c14d9ca26b2a9b2e921d50 P 8d3f261588f14e158cc4b8a3388f5136 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8d3f261588f14e158cc4b8a3388f5136; no voters: a87fb8ce1b304e8e9dbc3e969e690d9b, fa3486d01e9045dcba230304040c37f1
I20260619 18:50:27.478331 19500 raft_consensus.cc:2726] T 485ff9db72c14d9ca26b2a9b2e921d50 P 8d3f261588f14e158cc4b8a3388f5136 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 18:50:27.870748 19501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:27.870055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:39697 (local address 127.3.243.67:49877)
0619 18:50:27.870507 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:27.870593 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:39697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":211,"threads_started":1}
I20260619 18:50:28.046198 19502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:28.045472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:39697 (local address 127.3.243.65:48519)
0619 18:50:28.045930 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:28.046027 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:39697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":234,"threads_started":1}
I20260619 18:50:28.451483 19503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:28.450818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:39697 (local address 127.3.243.66:57689)
0619 18:50:28.451229 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:28.451303 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:39697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":215,"threads_started":1}
I20260619 18:50:28.872431 19504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:28.871589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:39697 (local address 127.3.243.67:41303)
0619 18:50:28.872067 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:28.872191 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:39697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":250,"threads_started":1}
I20260619 18:50:29.047667 19505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:29.046966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:39697 (local address 127.3.243.65:34535)
0619 18:50:29.047411 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:29.047499 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:39697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":234,"threads_started":1}
I20260619 18:50:29.453152 19506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:29.452438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:39697 (local address 127.3.243.66:47405)
0619 18:50:29.452891 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:29.452996 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:39697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":211,"threads_started":1}
I20260619 18:50:29.688879 19507 raft_consensus.cc:482] T 485ff9db72c14d9ca26b2a9b2e921d50 P 8d3f261588f14e158cc4b8a3388f5136 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:50:29.689056 19507 raft_consensus.cc:504] T 485ff9db72c14d9ca26b2a9b2e921d50 P 8d3f261588f14e158cc4b8a3388f5136 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d3f261588f14e158cc4b8a3388f5136" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 46855 } } peers { permanent_uuid: "fa3486d01e9045dcba230304040c37f1" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 46229 } } peers { permanent_uuid: "a87fb8ce1b304e8e9dbc3e969e690d9b" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 34767 } }
I20260619 18:50:29.689599 19507 leader_election.cc:288] T 485ff9db72c14d9ca26b2a9b2e921d50 P 8d3f261588f14e158cc4b8a3388f5136 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fa3486d01e9045dcba230304040c37f1 (127.3.243.66:46229), a87fb8ce1b304e8e9dbc3e969e690d9b (127.3.243.67:34767)
W20260619 18:50:29.690512 19081 leader_election.cc:341] T 485ff9db72c14d9ca26b2a9b2e921d50 P 8d3f261588f14e158cc4b8a3388f5136 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a87fb8ce1b304e8e9dbc3e969e690d9b (127.3.243.67:34767): Not found: Tablet not found: 485ff9db72c14d9ca26b2a9b2e921d50
W20260619 18:50:29.690703 19081 leader_election.cc:341] T 485ff9db72c14d9ca26b2a9b2e921d50 P 8d3f261588f14e158cc4b8a3388f5136 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fa3486d01e9045dcba230304040c37f1 (127.3.243.66:46229): Not found: Tablet not found: 485ff9db72c14d9ca26b2a9b2e921d50
I20260619 18:50:29.690812 19081 leader_election.cc:302] T 485ff9db72c14d9ca26b2a9b2e921d50 P 8d3f261588f14e158cc4b8a3388f5136 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8d3f261588f14e158cc4b8a3388f5136; no voters: a87fb8ce1b304e8e9dbc3e969e690d9b, fa3486d01e9045dcba230304040c37f1
I20260619 18:50:29.691015 19507 raft_consensus.cc:2726] T 485ff9db72c14d9ca26b2a9b2e921d50 P 8d3f261588f14e158cc4b8a3388f5136 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 18:50:29.873870 19508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:29.873213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:39697 (local address 127.3.243.67:40367)
0619 18:50:29.873616 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:29.873703 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:39697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":200,"threads_started":1}
I20260619 18:50:30.049154 19509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:30.048432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:39697 (local address 127.3.243.65:51687)
0619 18:50:30.048848 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:30.048976 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:39697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":216,"threads_started":1}
I20260619 18:50:30.454592 19510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:30.453896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:39697 (local address 127.3.243.66:34799)
0619 18:50:30.454333 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:30.454412 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:39697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":210,"threads_started":1}
I20260619 18:50:30.875324 19511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:30.874615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:39697 (local address 127.3.243.67:35455)
0619 18:50:30.875056 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:30.875135 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:39697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":228,"threads_started":1}
I20260619 18:50:31.050789 19512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:31.050121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:39697 (local address 127.3.243.65:48249)
0619 18:50:31.050532 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:31.050621 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:39697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":204,"threads_started":1}
I20260619 18:50:31.456260 19513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:31.455534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:39697 (local address 127.3.243.66:45439)
0619 18:50:31.455960 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:31.456062 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:39697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":207,"threads_started":1}
I20260619 18:50:31.862115 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 19032
I20260619 18:50:31.876986 19515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:31.876311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:39697 (local address 127.3.243.67:47849)
0619 18:50:31.876729 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:31.876840 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:39697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":228,"threads_started":1}
I20260619 18:50:31.882781 19514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:31.863045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36244 (local address 127.3.243.65:46855)
0619 18:50:31.863571 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0619 18:50:31.863580 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:50:31.863617 (+ 37us) server_negotiation.cc:407] Connection header received
0619 18:50:31.863849 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:50:31.863859 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:50:31.864174 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:50:31.864355 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:50:31.865883 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:31.866828 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:31.867923 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:31.868502 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:31.868598 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:31.877849 (+ 9251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:50:31.877860 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:50:31.877875 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:50:31.877911 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:50:31.880428 (+ 2517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:31.880999 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:31.881006 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:31.881008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:31.881137 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:31.881413 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:31.881416 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:31.881418 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:31.881816 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:50:31.882047 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0619 18:50:31.882264 (+ 217us) server_negotiation.cc:299] Negotiation successful
0619 18:50:31.882470 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":206,"threads_started":1}
I20260619 18:50:31.883932 19189 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:50:31.968582 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 19032
I20260619 18:50:31.982255 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 19198
I20260619 18:50:31.999559 19517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:31.982935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50842 (local address 127.3.243.66:46229)
0619 18:50:31.983382 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0619 18:50:31.983386 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:50:31.983401 (+ 15us) server_negotiation.cc:407] Connection header received
0619 18:50:31.983454 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:50:31.983458 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:50:31.983515 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:50:31.983598 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:50:31.984922 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:31.985644 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:31.986904 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:31.987316 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:31.987403 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:31.995040 (+ 7637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:50:31.995045 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:50:31.995048 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:50:31.995097 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:50:31.997500 (+ 2403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:31.998121 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:31.998124 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:31.998126 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:31.998235 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:31.998490 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:31.998493 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:31.998494 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:31.998735 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:50:31.998930 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0619 18:50:31.999122 (+ 192us) server_negotiation.cc:299] Negotiation successful
0619 18:50:31.999278 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":229,"threads_started":1}
I20260619 18:50:32.000353 19321 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:50:32.080168 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 19198
I20260619 18:50:32.094879 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 19329
I20260619 18:50:32.107532 19519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:32.095712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57690 (local address 127.3.243.67:34767)
0619 18:50:32.096294 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0619 18:50:32.096299 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:50:32.096317 (+ 18us) server_negotiation.cc:407] Connection header received
0619 18:50:32.096379 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:50:32.096383 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:50:32.096451 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:50:32.096537 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:50:32.098011 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:32.098793 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:32.099922 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:32.100365 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:32.100456 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:32.102987 (+ 2531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:50:32.102992 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:50:32.102995 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:50:32.103018 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:50:32.105388 (+ 2370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:32.106032 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:32.106035 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:32.106037 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:32.106164 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:32.106441 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:32.106444 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:32.106445 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:32.106662 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:50:32.106847 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0619 18:50:32.107089 (+ 242us) server_negotiation.cc:299] Negotiation successful
0619 18:50:32.107239 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":293,"threads_started":1}
I20260619 18:50:32.108477 19452 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:50:32.197476 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 19329
2026-06-19T18:50:32Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (23862 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:50:32 dist-test-slave-m9wg krb5kdc[19523](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:50:32 dist-test-slave-m9wg krb5kdc[19523](info): set up 2 sockets
Jun 19 18:50:32 dist-test-slave-m9wg krb5kdc[19523](info): commencing operation
krb5kdc: starting...
W20260619 18:50:33.224460 4045 mini_kdc.cc:121] Time spent starting KDC: real 0.991s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/krb5kdc/test-user.keytab.
Jun 19 18:50:33 dist-test-slave-m9wg krb5kdc[19523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895033, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:50:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:50:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.243.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab.
Entry for principal kudu/127.3.243.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab.
I20260619 18:50:33.445482 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.126:42879
--webserver_interface=127.3.243.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab
--principal=kudu/127.3.243.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:45379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.126:42879
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:50:33.755514 19536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:50:33.755800 19536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:50:33.755934 19536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:50:33.762849 19536 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 18:50:33.762948 19536 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 18:50:33.763007 19536 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 18:50:33.763082 19536 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:50:33.763154 19536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:50:33.763253 19536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:50:33.763309 19536 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:50:33.763382 19536 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:50:33.772909 19536 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:45379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.3.243.126:42879
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab
--principal=kudu/127.3.243.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.126:42879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.243.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:50:33.774135 19536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:50:33.775452 19536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:50:33.786706 19541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:50:33.786681 19544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:50:33.786636 19542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:50:33.787278 19536 server_base.cc:1034] running on GCE node
I20260619 18:50:33.788030 19536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:50:33.789397 19536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:50:33.790606 19536 hybrid_clock.cc:648] HybridClock initialized: now 1781895033790545 us; error 34 us; skew 500 ppm
I20260619 18:50:33.790911 19536 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:50:33 dist-test-slave-m9wg krb5kdc[19523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895033, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:50:33.793869 19536 init.cc:377] Logged in from keytab as kudu/127.3.243.126@KRBTEST.COM (short username kudu)
I20260619 18:50:33.795249 19536 webserver.cc:460] Webserver started at http://127.3.243.126:46413/ using document root <none> and password file <none>
I20260619 18:50:33.795893 19536 fs_manager.cc:359] Metadata directory not provided
I20260619 18:50:33.795987 19536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:50:33.796320 19536 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:50:33.799145 19536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/master-0/data/instance:
uuid: "5eb6468c42124bb8acf48ba3fbceac41"
format_stamp: "Formatted at 2026-06-19 18:50:33 on dist-test-slave-m9wg"
server_key: "5bee5c99c932f10c288f5b8278cc5e63"
server_key_iv: "9b5ff271c13b1f6b55a7ae9a06a2e25f"
server_key_version: "encryptionkey@0"
I20260619 18:50:33.799880 19536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/master-0/wal/instance:
uuid: "5eb6468c42124bb8acf48ba3fbceac41"
format_stamp: "Formatted at 2026-06-19 18:50:33 on dist-test-slave-m9wg"
server_key: "5bee5c99c932f10c288f5b8278cc5e63"
server_key_iv: "9b5ff271c13b1f6b55a7ae9a06a2e25f"
server_key_version: "encryptionkey@0"
I20260619 18:50:33.804726 19536 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260619 18:50:33.808018 19551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:50:33.809736 19536 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:50:33.809896 19536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/master-0/wal
uuid: "5eb6468c42124bb8acf48ba3fbceac41"
format_stamp: "Formatted at 2026-06-19 18:50:33 on dist-test-slave-m9wg"
server_key: "5bee5c99c932f10c288f5b8278cc5e63"
server_key_iv: "9b5ff271c13b1f6b55a7ae9a06a2e25f"
server_key_version: "encryptionkey@0"
I20260619 18:50:33.810108 19536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:50:33.851125 19536 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:50:33.855139 19536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:50:33.855386 19536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:50:33.872208 19536 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.126:42879
I20260619 18:50:33.872205 19602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.126:42879 every 8 connection(s)
I20260619 18:50:33.873585 19536 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
I20260619 18:50:33.874977 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 19536
I20260619 18:50:33.875187 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/master-0/wal/instance
I20260619 18:50:33.875991 4045 external_mini_cluster.cc:1428] Setting key 71c476b3e318db2602a571a852e67449
I20260619 18:50:33.878495 19603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 18:50:33 dist-test-slave-m9wg krb5kdc[19523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895033, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.126@KRBTEST.COM
I20260619 18:50:33.891227 19603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5eb6468c42124bb8acf48ba3fbceac41: Bootstrap starting.
I20260619 18:50:33.895272 19603 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5eb6468c42124bb8acf48ba3fbceac41: Neither blocks nor log segments found. Creating new log.
I20260619 18:50:33.896452 19603 log.cc:824] T 00000000000000000000000000000000 P 5eb6468c42124bb8acf48ba3fbceac41: Log is configured to *not* fsync() on all Append() calls
I20260619 18:50:33.899667 19606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:33.878103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40878 (local address 127.3.243.126:42879)
0619 18:50:33.878862 (+ 759us) server_negotiation.cc:206] Beginning negotiation
0619 18:50:33.878874 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 18:50:33.878913 (+ 39us) server_negotiation.cc:407] Connection header received
0619 18:50:33.880523 (+ 1610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:50:33.880546 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:50:33.880926 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:50:33.881238 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:50:33.882956 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:33.884305 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:33.886061 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:33.886696 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:33.886816 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:33.891494 (+ 4678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:50:33.891504 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:50:33.891522 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:50:33.891570 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:50:33.895210 (+ 3640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:33.895901 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:33.895909 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:33.895916 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:33.896121 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:33.896494 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:33.896498 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:33.896501 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:33.897197 (+ 696us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:50:33.897409 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0619 18:50:33.897948 (+ 539us) server_negotiation.cc:299] Negotiation successful
0619 18:50:33.898387 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":349,"threads_started":1}
I20260619 18:50:33.899745 19603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5eb6468c42124bb8acf48ba3fbceac41: No bootstrap required, opened a new log
I20260619 18:50:33.907209 19603 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5eb6468c42124bb8acf48ba3fbceac41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5eb6468c42124bb8acf48ba3fbceac41" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 42879 } }
I20260619 18:50:33.907536 19603 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5eb6468c42124bb8acf48ba3fbceac41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:50:33.907606 19603 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5eb6468c42124bb8acf48ba3fbceac41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5eb6468c42124bb8acf48ba3fbceac41, State: Initialized, Role: FOLLOWER
I20260619 18:50:33.908178 19603 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5eb6468c42124bb8acf48ba3fbceac41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5eb6468c42124bb8acf48ba3fbceac41" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 42879 } }
I20260619 18:50:33.908432 19603 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5eb6468c42124bb8acf48ba3fbceac41 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:50:33.908566 19603 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5eb6468c42124bb8acf48ba3fbceac41 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:50:33.908828 19603 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5eb6468c42124bb8acf48ba3fbceac41 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:50:33.910861 19603 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5eb6468c42124bb8acf48ba3fbceac41 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5eb6468c42124bb8acf48ba3fbceac41" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 42879 } }
I20260619 18:50:33.911327 19603 leader_election.cc:302] T 00000000000000000000000000000000 P 5eb6468c42124bb8acf48ba3fbceac41 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5eb6468c42124bb8acf48ba3fbceac41; no voters:
I20260619 18:50:33.911823 19603 leader_election.cc:288] T 00000000000000000000000000000000 P 5eb6468c42124bb8acf48ba3fbceac41 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:50:33.911944 19608 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5eb6468c42124bb8acf48ba3fbceac41 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:50:33.912544 19608 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5eb6468c42124bb8acf48ba3fbceac41 [term 1 LEADER]: Becoming Leader. State: Replica: 5eb6468c42124bb8acf48ba3fbceac41, State: Running, Role: LEADER
I20260619 18:50:33.913089 19608 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5eb6468c42124bb8acf48ba3fbceac41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5eb6468c42124bb8acf48ba3fbceac41" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 42879 } }
I20260619 18:50:33.913862 19603 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5eb6468c42124bb8acf48ba3fbceac41 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:50:33.915935 19609 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5eb6468c42124bb8acf48ba3fbceac41 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5eb6468c42124bb8acf48ba3fbceac41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5eb6468c42124bb8acf48ba3fbceac41" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 42879 } } }
I20260619 18:50:33.916313 19609 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5eb6468c42124bb8acf48ba3fbceac41 [sys.catalog]: This master's current role is: LEADER
I20260619 18:50:33.916006 19610 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5eb6468c42124bb8acf48ba3fbceac41 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5eb6468c42124bb8acf48ba3fbceac41. Latest consensus state: current_term: 1 leader_uuid: "5eb6468c42124bb8acf48ba3fbceac41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5eb6468c42124bb8acf48ba3fbceac41" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 42879 } } }
I20260619 18:50:33.916564 19610 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5eb6468c42124bb8acf48ba3fbceac41 [sys.catalog]: This master's current role is: LEADER
I20260619 18:50:33.918146 19613 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:50:33.923882 19613 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:50:33.932487 19613 catalog_manager.cc:1347] Generated new cluster ID: c15cbeedde704a27b2d2b3bbd371d776
I20260619 18:50:33.932574 19613 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:50:33.950464 19613 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:50:33.951689 19613 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:50:33.959940 19613 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5eb6468c42124bb8acf48ba3fbceac41: Generated new TSK 0
I20260619 18:50:33.960700 19613 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.243.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.65.keytab.
Entry for principal kudu/127.3.243.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.65.keytab.
I20260619 18:50:34.013795 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.65:0
--local_ip_for_outbound_sockets=127.3.243.65
--webserver_interface=127.3.243.65
--webserver_port=0
--tserver_master_addrs=127.3.243.126:42879
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.65.keytab
--principal=kudu/127.3.243.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:45379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:50:34.325479 19629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:50:34.325743 19629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:50:34.325875 19629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:50:34.332774 19629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:50:34.332862 19629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:50:34.333001 19629 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.65
I20260619 18:50:34.342082 19629 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:45379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.65.keytab
--principal=kudu/127.3.243.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.243.65
--webserver_port=0
--tserver_master_addrs=127.3.243.126:42879
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.65
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:50:34.343228 19629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:50:34.344496 19629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:50:34.356271 19635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:50:34.356326 19637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:50:34.356544 19634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:50:34.357409 19629 server_base.cc:1034] running on GCE node
I20260619 18:50:34.357983 19629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:50:34.358903 19629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:50:34.360102 19629 hybrid_clock.cc:648] HybridClock initialized: now 1781895034360057 us; error 41 us; skew 500 ppm
I20260619 18:50:34.360402 19629 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:50:34 dist-test-slave-m9wg krb5kdc[19523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895034, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:50:34.363268 19629 init.cc:377] Logged in from keytab as kudu/127.3.243.65@KRBTEST.COM (short username kudu)
I20260619 18:50:34.364635 19629 webserver.cc:460] Webserver started at http://127.3.243.65:34305/ using document root <none> and password file <none>
I20260619 18:50:34.365259 19629 fs_manager.cc:359] Metadata directory not provided
I20260619 18:50:34.365371 19629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:50:34.365666 19629 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:50:34.368398 19629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-0/data/instance:
uuid: "4c5611deb12845d982674dcfea33fd93"
format_stamp: "Formatted at 2026-06-19 18:50:34 on dist-test-slave-m9wg"
server_key: "8949a1fc82148059d2455b1dfd58e8bb"
server_key_iv: "eb1da66fec32d889e1dc8f419c86652b"
server_key_version: "encryptionkey@0"
I20260619 18:50:34.369118 19629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-0/wal/instance:
uuid: "4c5611deb12845d982674dcfea33fd93"
format_stamp: "Formatted at 2026-06-19 18:50:34 on dist-test-slave-m9wg"
server_key: "8949a1fc82148059d2455b1dfd58e8bb"
server_key_iv: "eb1da66fec32d889e1dc8f419c86652b"
server_key_version: "encryptionkey@0"
I20260619 18:50:34.373725 19629 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 18:50:34.376865 19644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:50:34.378032 19629 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 18:50:34.378170 19629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-0/wal
uuid: "4c5611deb12845d982674dcfea33fd93"
format_stamp: "Formatted at 2026-06-19 18:50:34 on dist-test-slave-m9wg"
server_key: "8949a1fc82148059d2455b1dfd58e8bb"
server_key_iv: "eb1da66fec32d889e1dc8f419c86652b"
server_key_version: "encryptionkey@0"
I20260619 18:50:34.378345 19629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:50:34.390381 19629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:50:34.393757 19629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:50:34.393990 19629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:50:34.394810 19629 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:50:34.395993 19629 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:50:34.396070 19629 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:50:34.396215 19629 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:50:34.396270 19629 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260619 18:50:34.425587 19629 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.65:38277
I20260619 18:50:34.425627 19756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.65:38277 every 8 connection(s)
I20260619 18:50:34.426769 19629 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-0/data/info.pb
I20260619 18:50:34.436823 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 19629
I20260619 18:50:34.437021 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-0/wal/instance
I20260619 18:50:34.437682 4045 external_mini_cluster.cc:1428] Setting key a3638bd6a83eaa73f86f7137d772c291
Jun 19 18:50:34 dist-test-slave-m9wg krb5kdc[19523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895034, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.65@KRBTEST.COM for kudu/127.3.243.126@KRBTEST.COM
I20260619 18:50:34.445147 19759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:34.429048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.65:50845 (local address 127.3.243.126:42879)
0619 18:50:34.429447 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0619 18:50:34.429452 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:50:34.430329 (+ 877us) server_negotiation.cc:407] Connection header received
0619 18:50:34.432058 (+ 1729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:50:34.432063 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:50:34.432139 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:50:34.432271 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:50:34.433677 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:34.434398 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:34.435857 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:34.436290 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:34.436400 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:34.439890 (+ 3490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:50:34.439895 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:50:34.439898 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:50:34.439927 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:50:34.442288 (+ 2361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:34.442944 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:34.442949 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:34.442953 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:34.443072 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:34.443462 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:34.443467 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:34.443471 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:34.443735 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:50:34.443859 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 18:50:34.444561 (+ 702us) server_negotiation.cc:299] Negotiation successful
0619 18:50:34.444786 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":201,"threads_started":1}
I20260619 18:50:34.446060 19760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:34.429521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:42879 (local address 127.3.243.65:50845)
0619 18:50:34.430169 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:34.430209 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 18:50:34.431795 (+ 1586us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:50:34.432498 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:50:34.432506 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:50:34.432992 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:50:34.433515 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:50:34.433532 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:50:34.434536 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:50:34.434540 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:50:34.435719 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:50:34.435735 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:50:34.436506 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:50:34.436514 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:50:34.436732 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:34.437381 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:50:34.437401 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:50:34.439694 (+ 2293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:50:34.442418 (+ 2724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:50:34.442426 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:50:34.442442 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:50:34.442839 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:50:34.443178 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:50:34.443183 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:50:34.443187 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:50:34.443360 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:50:34.443868 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:50:34.443875 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:50:34.444229 (+ 354us) client_negotiation.cc:769] Sending connection context
0619 18:50:34.444470 (+ 241us) client_negotiation.cc:240] Negotiation successful
0619 18:50:34.444981 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":256,"threads_started":1}
I20260619 18:50:34.447494 19757 heartbeater.cc:347] Connected to a master server at 127.3.243.126:42879
I20260619 18:50:34.447768 19757 heartbeater.cc:464] Registering TS with master...
I20260619 18:50:34.448482 19757 heartbeater.cc:511] Master 127.3.243.126:42879 requested a full tablet report, sending...
I20260619 18:50:34.450627 19568 ts_manager.cc:194] Registered new tserver with Master: 4c5611deb12845d982674dcfea33fd93 (127.3.243.65:38277)
I20260619 18:50:34.452630 19568 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.65@KRBTEST.COM'} at 127.3.243.65:50845
WARNING: no policy specified for kudu/127.3.243.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.66.keytab.
Entry for principal kudu/127.3.243.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.66.keytab.
I20260619 18:50:34.484719 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.66:0
--local_ip_for_outbound_sockets=127.3.243.66
--webserver_interface=127.3.243.66
--webserver_port=0
--tserver_master_addrs=127.3.243.126:42879
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.66.keytab
--principal=kudu/127.3.243.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:45379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:50:34.788090 19763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:50:34.788395 19763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:50:34.788533 19763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:50:34.795445 19763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:50:34.795537 19763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:50:34.795681 19763 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.66
I20260619 18:50:34.804689 19763 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:45379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.66.keytab
--principal=kudu/127.3.243.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.243.66
--webserver_port=0
--tserver_master_addrs=127.3.243.126:42879
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.66
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:50:34.805799 19763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:50:34.806990 19763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:50:34.819595 19768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:50:34.819630 19771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:50:34.819649 19769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:50:34.820662 19763 server_base.cc:1034] running on GCE node
I20260619 18:50:34.821174 19763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:50:34.822028 19763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:50:34.823220 19763 hybrid_clock.cc:648] HybridClock initialized: now 1781895034823181 us; error 28 us; skew 500 ppm
I20260619 18:50:34.823494 19763 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:50:34 dist-test-slave-m9wg krb5kdc[19523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895034, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:50:34.826258 19763 init.cc:377] Logged in from keytab as kudu/127.3.243.66@KRBTEST.COM (short username kudu)
I20260619 18:50:34.827531 19763 webserver.cc:460] Webserver started at http://127.3.243.66:36089/ using document root <none> and password file <none>
I20260619 18:50:34.828130 19763 fs_manager.cc:359] Metadata directory not provided
I20260619 18:50:34.828274 19763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:50:34.828560 19763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:50:34.831245 19763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-1/data/instance:
uuid: "2ab38de74582419e893b702c9dd7a051"
format_stamp: "Formatted at 2026-06-19 18:50:34 on dist-test-slave-m9wg"
server_key: "2e62b509e7add8e89f6a640ab71367ab"
server_key_iv: "4bdd0a384976c71967d5e44491c4d209"
server_key_version: "encryptionkey@0"
I20260619 18:50:34.831946 19763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-1/wal/instance:
uuid: "2ab38de74582419e893b702c9dd7a051"
format_stamp: "Formatted at 2026-06-19 18:50:34 on dist-test-slave-m9wg"
server_key: "2e62b509e7add8e89f6a640ab71367ab"
server_key_iv: "4bdd0a384976c71967d5e44491c4d209"
server_key_version: "encryptionkey@0"
I20260619 18:50:34.836680 19763 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260619 18:50:34.839787 19778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:50:34.841022 19763 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260619 18:50:34.841162 19763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-1/wal
uuid: "2ab38de74582419e893b702c9dd7a051"
format_stamp: "Formatted at 2026-06-19 18:50:34 on dist-test-slave-m9wg"
server_key: "2e62b509e7add8e89f6a640ab71367ab"
server_key_iv: "4bdd0a384976c71967d5e44491c4d209"
server_key_version: "encryptionkey@0"
I20260619 18:50:34.841342 19763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:50:34.853513 19763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:50:34.856848 19763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:50:34.857082 19763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:50:34.857884 19763 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:50:34.859032 19763 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:50:34.859105 19763 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:50:34.859223 19763 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:50:34.859282 19763 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:50:34.888633 19763 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.66:37273
I20260619 18:50:34.888669 19890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.66:37273 every 8 connection(s)
I20260619 18:50:34.889780 19763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-1/data/info.pb
I20260619 18:50:34.898085 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 19763
I20260619 18:50:34.898258 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-1/wal/instance
I20260619 18:50:34.898695 4045 external_mini_cluster.cc:1428] Setting key 04489f23cd87f2c2b5404e209d394d81
Jun 19 18:50:34 dist-test-slave-m9wg krb5kdc[19523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895034, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.66@KRBTEST.COM for kudu/127.3.243.126@KRBTEST.COM
I20260619 18:50:34.908390 19759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:34.892076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.66:52979 (local address 127.3.243.126:42879)
0619 18:50:34.892276 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 18:50:34.892280 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:50:34.893385 (+ 1105us) server_negotiation.cc:407] Connection header received
0619 18:50:34.895060 (+ 1675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:50:34.895064 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:50:34.895129 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:50:34.895212 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:50:34.896582 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:34.897264 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:34.898772 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:34.899326 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:34.899428 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:34.902690 (+ 3262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:50:34.902696 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:50:34.902700 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:50:34.902738 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:50:34.905579 (+ 2841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:34.906227 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:34.906232 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:34.906235 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:34.906406 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:34.906764 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:34.906768 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:34.906772 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:34.907079 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:50:34.907216 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 18:50:34.907830 (+ 614us) server_negotiation.cc:299] Negotiation successful
0619 18:50:34.908046 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 18:50:34.909192 19893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:34.892571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:42879 (local address 127.3.243.66:52979)
0619 18:50:34.893223 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:34.893259 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 18:50:34.894831 (+ 1572us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:50:34.895389 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:50:34.895397 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:50:34.895884 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:50:34.896456 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:50:34.896474 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:50:34.897375 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:50:34.897378 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:50:34.898625 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:50:34.898636 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:50:34.899563 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:50:34.899569 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:50:34.899778 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:34.900467 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:50:34.900489 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:50:34.902494 (+ 2005us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:50:34.905703 (+ 3209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:50:34.905710 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:50:34.905722 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:50:34.906115 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:50:34.906493 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:50:34.906496 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:50:34.906498 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:50:34.906665 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:50:34.907176 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:50:34.907182 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:50:34.907490 (+ 308us) client_negotiation.cc:769] Sending connection context
0619 18:50:34.907704 (+ 214us) client_negotiation.cc:240] Negotiation successful
0619 18:50:34.908078 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":249,"threads_started":1}
I20260619 18:50:34.910653 19891 heartbeater.cc:347] Connected to a master server at 127.3.243.126:42879
I20260619 18:50:34.910950 19891 heartbeater.cc:464] Registering TS with master...
I20260619 18:50:34.911653 19891 heartbeater.cc:511] Master 127.3.243.126:42879 requested a full tablet report, sending...
I20260619 18:50:34.913103 19568 ts_manager.cc:194] Registered new tserver with Master: 2ab38de74582419e893b702c9dd7a051 (127.3.243.66:37273)
I20260619 18:50:34.914197 19568 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.66@KRBTEST.COM'} at 127.3.243.66:52979
WARNING: no policy specified for kudu/127.3.243.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.67.keytab.
Entry for principal kudu/127.3.243.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.67.keytab.
I20260619 18:50:34.949321 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.67:0
--local_ip_for_outbound_sockets=127.3.243.67
--webserver_interface=127.3.243.67
--webserver_port=0
--tserver_master_addrs=127.3.243.126:42879
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.67.keytab
--principal=kudu/127.3.243.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:45379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:50:35.256336 19896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:50:35.256597 19896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:50:35.256727 19896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:50:35.263604 19896 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:50:35.263705 19896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:50:35.263849 19896 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.67
I20260619 18:50:35.273118 19896 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:45379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.67.keytab
--principal=kudu/127.3.243.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.243.67
--webserver_port=0
--tserver_master_addrs=127.3.243.126:42879
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.67
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:50:35.274329 19896 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:50:35.275614 19896 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:50:35.287704 19901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:50:35.287763 19904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:50:35.287755 19902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:50:35.288358 19896 server_base.cc:1034] running on GCE node
I20260619 18:50:35.288919 19896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:50:35.289788 19896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:50:35.290984 19896 hybrid_clock.cc:648] HybridClock initialized: now 1781895035290938 us; error 39 us; skew 500 ppm
I20260619 18:50:35.291251 19896 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:50:35 dist-test-slave-m9wg krb5kdc[19523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895035, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:50:35.294034 19896 init.cc:377] Logged in from keytab as kudu/127.3.243.67@KRBTEST.COM (short username kudu)
I20260619 18:50:35.295364 19896 webserver.cc:460] Webserver started at http://127.3.243.67:45589/ using document root <none> and password file <none>
I20260619 18:50:35.295987 19896 fs_manager.cc:359] Metadata directory not provided
I20260619 18:50:35.296097 19896 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:50:35.296396 19896 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:50:35.299091 19896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-2/data/instance:
uuid: "bad6f9f3febb40e8b9161ba802d06542"
format_stamp: "Formatted at 2026-06-19 18:50:35 on dist-test-slave-m9wg"
server_key: "3cedacdc2802ab3a8ff6cf01cef22bf4"
server_key_iv: "d46435461ac1345cae481d52c8f3b7e8"
server_key_version: "encryptionkey@0"
I20260619 18:50:35.299780 19896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-2/wal/instance:
uuid: "bad6f9f3febb40e8b9161ba802d06542"
format_stamp: "Formatted at 2026-06-19 18:50:35 on dist-test-slave-m9wg"
server_key: "3cedacdc2802ab3a8ff6cf01cef22bf4"
server_key_iv: "d46435461ac1345cae481d52c8f3b7e8"
server_key_version: "encryptionkey@0"
I20260619 18:50:35.304584 19896 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260619 18:50:35.307708 19911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:50:35.308981 19896 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:50:35.309118 19896 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-2/wal
uuid: "bad6f9f3febb40e8b9161ba802d06542"
format_stamp: "Formatted at 2026-06-19 18:50:35 on dist-test-slave-m9wg"
server_key: "3cedacdc2802ab3a8ff6cf01cef22bf4"
server_key_iv: "d46435461ac1345cae481d52c8f3b7e8"
server_key_version: "encryptionkey@0"
I20260619 18:50:35.309281 19896 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:50:35.328744 19896 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:50:35.332331 19896 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:50:35.332567 19896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:50:35.333461 19896 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:50:35.334692 19896 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:50:35.334766 19896 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:50:35.334890 19896 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:50:35.334945 19896 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:50:35.364230 19896 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.67:40449
I20260619 18:50:35.364277 20023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.67:40449 every 8 connection(s)
I20260619 18:50:35.365526 19896 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-2/data/info.pb
I20260619 18:50:35.369443 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 19896
I20260619 18:50:35.369637 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-2/wal/instance
I20260619 18:50:35.370200 4045 external_mini_cluster.cc:1428] Setting key 16c786f602288110a5dce52be4d801de
Jun 19 18:50:35 dist-test-slave-m9wg krb5kdc[19523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895035, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.67@KRBTEST.COM for kudu/127.3.243.126@KRBTEST.COM
I20260619 18:50:35.384121 19759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:35.367912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.67:56941 (local address 127.3.243.126:42879)
0619 18:50:35.368087 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 18:50:35.368092 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:50:35.369233 (+ 1141us) server_negotiation.cc:407] Connection header received
0619 18:50:35.371527 (+ 2294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:50:35.371532 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:50:35.371596 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:50:35.371671 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:50:35.373377 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:35.374090 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:35.375544 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:35.375966 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:35.376050 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:35.379184 (+ 3134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:50:35.379189 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:50:35.379192 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:50:35.379223 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:50:35.381514 (+ 2291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:35.382124 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:35.382127 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:35.382129 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:35.382234 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:35.382584 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:35.382587 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:35.382589 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:35.382844 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:50:35.382948 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 18:50:35.383526 (+ 578us) server_negotiation.cc:299] Negotiation successful
0619 18:50:35.383750 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 18:50:35.385187 20026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:35.368365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:42879 (local address 127.3.243.67:56941)
0619 18:50:35.369034 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:35.369098 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0619 18:50:35.371259 (+ 2161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:50:35.371936 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:50:35.371947 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:50:35.372539 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:50:35.373213 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:50:35.373245 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:50:35.374223 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:50:35.374228 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:50:35.375442 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:50:35.375454 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:50:35.376136 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:50:35.376141 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:50:35.376441 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:35.377022 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:50:35.377047 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:50:35.379019 (+ 1972us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:50:35.381632 (+ 2613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:50:35.381638 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:50:35.381648 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:50:35.382039 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:50:35.382318 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:50:35.382323 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:50:35.382326 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:50:35.382497 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:50:35.382927 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:50:35.382932 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:50:35.383196 (+ 264us) client_negotiation.cc:769] Sending connection context
0619 18:50:35.383471 (+ 275us) client_negotiation.cc:240] Negotiation successful
0619 18:50:35.383947 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":246,"threads_started":1}
I20260619 18:50:35.386322 20024 heartbeater.cc:347] Connected to a master server at 127.3.243.126:42879
I20260619 18:50:35.386593 20024 heartbeater.cc:464] Registering TS with master...
I20260619 18:50:35.387244 20024 heartbeater.cc:511] Master 127.3.243.126:42879 requested a full tablet report, sending...
I20260619 18:50:35.388501 19568 ts_manager.cc:194] Registered new tserver with Master: bad6f9f3febb40e8b9161ba802d06542 (127.3.243.67:40449)
I20260619 18:50:35.389199 19568 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.67@KRBTEST.COM'} at 127.3.243.67:56941
I20260619 18:50:35.397073 4045 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 18:50:35.412601 19759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:35.401320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40886 (local address 127.3.243.126:42879)
0619 18:50:35.401479 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 18:50:35.401484 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:50:35.401867 (+ 383us) server_negotiation.cc:407] Connection header received
0619 18:50:35.401997 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:50:35.402000 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:50:35.402059 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:50:35.402133 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:50:35.403423 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:35.404115 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:35.405453 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:35.405877 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:35.405975 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:35.407955 (+ 1980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:50:35.407959 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:50:35.407963 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:50:35.407993 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:50:35.410444 (+ 2451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:35.410977 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:35.410981 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:35.410983 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:35.411102 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:35.411367 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:35.411370 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:35.411372 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:35.411703 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:50:35.411879 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0619 18:50:35.412193 (+ 314us) server_negotiation.cc:299] Negotiation successful
0619 18:50:35.412378 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 18:50:35.416292 19568 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40886:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:50:35.423378 19568 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:50:35.455510 19757 heartbeater.cc:503] Master 127.3.243.126:42879 was elected leader, sending a full tablet report...
I20260619 18:50:35.460232 20037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:35.434836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.67:40449 (local address 127.0.0.1:34600)
0619 18:50:35.445904 (+ 11068us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:35.445937 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 18:50:35.446090 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:50:35.451975 (+ 5885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:50:35.451983 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:50:35.452013 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:50:35.452523 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:50:35.452539 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:50:35.454929 (+ 2390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:50:35.454934 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:50:35.457329 (+ 2395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:50:35.457341 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:50:35.459210 (+ 1869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:50:35.459216 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:50:35.459421 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:35.459474 (+ 53us) client_negotiation.cc:769] Sending connection context
0619 18:50:35.459608 (+ 134us) client_negotiation.cc:240] Negotiation successful
0619 18:50:35.459785 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10834,"mutex_wait_us":89,"thread_start_us":17146,"threads_started":1}
I20260619 18:50:35.460284 20038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:35.435091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34600 (local address 127.3.243.67:40449)
0619 18:50:35.451201 (+ 16110us) server_negotiation.cc:206] Beginning negotiation
0619 18:50:35.451210 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:50:35.451247 (+ 37us) server_negotiation.cc:407] Connection header received
0619 18:50:35.451349 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:50:35.451357 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:50:35.451595 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:50:35.452733 (+ 1138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:50:35.453108 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:35.454744 (+ 1636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:35.457466 (+ 2722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:35.458738 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:35.458854 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:35.459071 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0619 18:50:35.459741 (+ 670us) server_negotiation.cc:299] Negotiation successful
0619 18:50:35.459903 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15924,"thread_start_us":16610,"threads_started":1}
I20260619 18:50:35.460675 20040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:35.435152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.65:38277 (local address 127.0.0.1:35736)
0619 18:50:35.451600 (+ 16448us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:35.451619 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 18:50:35.451765 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:50:35.452441 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:50:35.452449 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:50:35.452474 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:50:35.452858 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:50:35.452870 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:50:35.454862 (+ 1992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:50:35.454867 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:50:35.457640 (+ 2773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:50:35.457655 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:50:35.459644 (+ 1989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:50:35.459650 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:50:35.459847 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:35.459879 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 18:50:35.459964 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 18:50:35.460074 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16324,"mutex_wait_us":194,"thread_start_us":10386,"threads_started":1}
I20260619 18:50:35.460675 20041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:35.437202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33116 (local address 127.3.243.66:37273)
0619 18:50:35.451184 (+ 13982us) server_negotiation.cc:206] Beginning negotiation
0619 18:50:35.451193 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:50:35.452414 (+ 1221us) server_negotiation.cc:407] Connection header received
0619 18:50:35.452692 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:50:35.452699 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:50:35.452917 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:50:35.453061 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:50:35.453681 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:35.454646 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:35.458008 (+ 3362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:35.459125 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:35.459250 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:35.459425 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0619 18:50:35.459991 (+ 566us) server_negotiation.cc:299] Negotiation successful
0619 18:50:35.460175 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13812,"thread_start_us":13673,"threads_started":1}
I20260619 18:50:35.461206 20036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:35.436742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.66:37273 (local address 127.0.0.1:33116)
0619 18:50:35.452314 (+ 15572us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:35.452329 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 18:50:35.452509 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:50:35.453262 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:50:35.453265 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:50:35.453283 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:50:35.453544 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:50:35.453551 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:50:35.454791 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:50:35.454795 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:50:35.457881 (+ 3086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:50:35.457890 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:50:35.459575 (+ 1685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:50:35.459581 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:50:35.459775 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:35.459825 (+ 50us) client_negotiation.cc:769] Sending connection context
0619 18:50:35.460758 (+ 933us) client_negotiation.cc:240] Negotiation successful
0619 18:50:35.460879 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15439,"mutex_wait_us":129,"thread_start_us":14591,"threads_started":1}
I20260619 18:50:35.462888 20039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:35.435276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35736 (local address 127.3.243.65:38277)
0619 18:50:35.450511 (+ 15235us) server_negotiation.cc:206] Beginning negotiation
0619 18:50:35.450521 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:50:35.451735 (+ 1214us) server_negotiation.cc:407] Connection header received
0619 18:50:35.451897 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:50:35.451904 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:50:35.452211 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:50:35.453050 (+ 839us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:50:35.453271 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:35.454758 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:35.457792 (+ 3034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:35.459156 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:35.459278 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:35.459486 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0619 18:50:35.460317 (+ 831us) server_negotiation.cc:299] Negotiation successful
0619 18:50:35.461175 (+ 858us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15055,"thread_start_us":15253,"threads_started":1}
I20260619 18:50:35.464193 19959 tablet_service.cc:1467] Processing CreateTablet for tablet 9658d380da634ccd98d12305a673e12b (DEFAULT_TABLE table=test-table [id=436e6219f0d749769602763645dddbbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:50:35.464617 19826 tablet_service.cc:1467] Processing CreateTablet for tablet 9658d380da634ccd98d12305a673e12b (DEFAULT_TABLE table=test-table [id=436e6219f0d749769602763645dddbbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:50:35.465725 19959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9658d380da634ccd98d12305a673e12b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:50:35.465821 19826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9658d380da634ccd98d12305a673e12b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:50:35.465889 19692 tablet_service.cc:1467] Processing CreateTablet for tablet 9658d380da634ccd98d12305a673e12b (DEFAULT_TABLE table=test-table [id=436e6219f0d749769602763645dddbbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:50:35.466945 19692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9658d380da634ccd98d12305a673e12b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:50:35.477365 20042 tablet_bootstrap.cc:492] T 9658d380da634ccd98d12305a673e12b P 4c5611deb12845d982674dcfea33fd93: Bootstrap starting.
I20260619 18:50:35.479442 20043 tablet_bootstrap.cc:492] T 9658d380da634ccd98d12305a673e12b P 2ab38de74582419e893b702c9dd7a051: Bootstrap starting.
I20260619 18:50:35.481397 20042 tablet_bootstrap.cc:654] T 9658d380da634ccd98d12305a673e12b P 4c5611deb12845d982674dcfea33fd93: Neither blocks nor log segments found. Creating new log.
I20260619 18:50:35.483088 20042 log.cc:824] T 9658d380da634ccd98d12305a673e12b P 4c5611deb12845d982674dcfea33fd93: Log is configured to *not* fsync() on all Append() calls
I20260619 18:50:35.483741 20043 tablet_bootstrap.cc:654] T 9658d380da634ccd98d12305a673e12b P 2ab38de74582419e893b702c9dd7a051: Neither blocks nor log segments found. Creating new log.
I20260619 18:50:35.485008 20043 log.cc:824] T 9658d380da634ccd98d12305a673e12b P 2ab38de74582419e893b702c9dd7a051: Log is configured to *not* fsync() on all Append() calls
I20260619 18:50:35.485776 20044 tablet_bootstrap.cc:492] T 9658d380da634ccd98d12305a673e12b P bad6f9f3febb40e8b9161ba802d06542: Bootstrap starting.
I20260619 18:50:35.486150 20042 tablet_bootstrap.cc:492] T 9658d380da634ccd98d12305a673e12b P 4c5611deb12845d982674dcfea33fd93: No bootstrap required, opened a new log
I20260619 18:50:35.486409 20042 ts_tablet_manager.cc:1397] T 9658d380da634ccd98d12305a673e12b P 4c5611deb12845d982674dcfea33fd93: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260619 18:50:35.487282 20043 tablet_bootstrap.cc:492] T 9658d380da634ccd98d12305a673e12b P 2ab38de74582419e893b702c9dd7a051: No bootstrap required, opened a new log
I20260619 18:50:35.487506 20043 ts_tablet_manager.cc:1397] T 9658d380da634ccd98d12305a673e12b P 2ab38de74582419e893b702c9dd7a051: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260619 18:50:35.488895 20044 tablet_bootstrap.cc:654] T 9658d380da634ccd98d12305a673e12b P bad6f9f3febb40e8b9161ba802d06542: Neither blocks nor log segments found. Creating new log.
I20260619 18:50:35.489985 20044 log.cc:824] T 9658d380da634ccd98d12305a673e12b P bad6f9f3febb40e8b9161ba802d06542: Log is configured to *not* fsync() on all Append() calls
I20260619 18:50:35.494505 20044 tablet_bootstrap.cc:492] T 9658d380da634ccd98d12305a673e12b P bad6f9f3febb40e8b9161ba802d06542: No bootstrap required, opened a new log
I20260619 18:50:35.494763 20044 ts_tablet_manager.cc:1397] T 9658d380da634ccd98d12305a673e12b P bad6f9f3febb40e8b9161ba802d06542: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.005s
I20260619 18:50:35.494540 20043 raft_consensus.cc:348] T 9658d380da634ccd98d12305a673e12b P 2ab38de74582419e893b702c9dd7a051 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bad6f9f3febb40e8b9161ba802d06542" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40449 } } peers { permanent_uuid: "4c5611deb12845d982674dcfea33fd93" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 38277 } } peers { permanent_uuid: "2ab38de74582419e893b702c9dd7a051" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 37273 } }
I20260619 18:50:35.494886 20043 raft_consensus.cc:374] T 9658d380da634ccd98d12305a673e12b P 2ab38de74582419e893b702c9dd7a051 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:50:35.494980 20043 raft_consensus.cc:729] T 9658d380da634ccd98d12305a673e12b P 2ab38de74582419e893b702c9dd7a051 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ab38de74582419e893b702c9dd7a051, State: Initialized, Role: FOLLOWER
I20260619 18:50:35.495488 20043 consensus_queue.cc:260] T 9658d380da634ccd98d12305a673e12b P 2ab38de74582419e893b702c9dd7a051 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bad6f9f3febb40e8b9161ba802d06542" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40449 } } peers { permanent_uuid: "4c5611deb12845d982674dcfea33fd93" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 38277 } } peers { permanent_uuid: "2ab38de74582419e893b702c9dd7a051" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 37273 } }
I20260619 18:50:35.496011 20042 raft_consensus.cc:348] T 9658d380da634ccd98d12305a673e12b P 4c5611deb12845d982674dcfea33fd93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bad6f9f3febb40e8b9161ba802d06542" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40449 } } peers { permanent_uuid: "4c5611deb12845d982674dcfea33fd93" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 38277 } } peers { permanent_uuid: "2ab38de74582419e893b702c9dd7a051" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 37273 } }
I20260619 18:50:35.496416 20042 raft_consensus.cc:374] T 9658d380da634ccd98d12305a673e12b P 4c5611deb12845d982674dcfea33fd93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:50:35.496511 20042 raft_consensus.cc:729] T 9658d380da634ccd98d12305a673e12b P 4c5611deb12845d982674dcfea33fd93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c5611deb12845d982674dcfea33fd93, State: Initialized, Role: FOLLOWER
I20260619 18:50:35.497054 20042 consensus_queue.cc:260] T 9658d380da634ccd98d12305a673e12b P 4c5611deb12845d982674dcfea33fd93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bad6f9f3febb40e8b9161ba802d06542" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40449 } } peers { permanent_uuid: "4c5611deb12845d982674dcfea33fd93" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 38277 } } peers { permanent_uuid: "2ab38de74582419e893b702c9dd7a051" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 37273 } }
I20260619 18:50:35.498744 20042 ts_tablet_manager.cc:1428] T 9658d380da634ccd98d12305a673e12b P 4c5611deb12845d982674dcfea33fd93: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260619 18:50:35.499670 19891 heartbeater.cc:503] Master 127.3.243.126:42879 was elected leader, sending a full tablet report...
I20260619 18:50:35.500224 20043 ts_tablet_manager.cc:1428] T 9658d380da634ccd98d12305a673e12b P 2ab38de74582419e893b702c9dd7a051: Time spent starting tablet: real 0.013s user 0.013s sys 0.001s
I20260619 18:50:35.504595 20044 raft_consensus.cc:348] T 9658d380da634ccd98d12305a673e12b P bad6f9f3febb40e8b9161ba802d06542 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bad6f9f3febb40e8b9161ba802d06542" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40449 } } peers { permanent_uuid: "4c5611deb12845d982674dcfea33fd93" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 38277 } } peers { permanent_uuid: "2ab38de74582419e893b702c9dd7a051" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 37273 } }
I20260619 18:50:35.504900 20044 raft_consensus.cc:374] T 9658d380da634ccd98d12305a673e12b P bad6f9f3febb40e8b9161ba802d06542 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:50:35.504987 20044 raft_consensus.cc:729] T 9658d380da634ccd98d12305a673e12b P bad6f9f3febb40e8b9161ba802d06542 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bad6f9f3febb40e8b9161ba802d06542, State: Initialized, Role: FOLLOWER
I20260619 18:50:35.505422 20044 consensus_queue.cc:260] T 9658d380da634ccd98d12305a673e12b P bad6f9f3febb40e8b9161ba802d06542 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bad6f9f3febb40e8b9161ba802d06542" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40449 } } peers { permanent_uuid: "4c5611deb12845d982674dcfea33fd93" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 38277 } } peers { permanent_uuid: "2ab38de74582419e893b702c9dd7a051" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 37273 } }
I20260619 18:50:35.506286 20024 heartbeater.cc:503] Master 127.3.243.126:42879 was elected leader, sending a full tablet report...
I20260619 18:50:35.506623 20044 ts_tablet_manager.cc:1428] T 9658d380da634ccd98d12305a673e12b P bad6f9f3febb40e8b9161ba802d06542: Time spent starting tablet: real 0.012s user 0.007s sys 0.005s
I20260619 18:50:35.545374 20050 raft_consensus.cc:482] T 9658d380da634ccd98d12305a673e12b P bad6f9f3febb40e8b9161ba802d06542 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:50:35.545651 20050 raft_consensus.cc:504] T 9658d380da634ccd98d12305a673e12b P bad6f9f3febb40e8b9161ba802d06542 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bad6f9f3febb40e8b9161ba802d06542" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40449 } } peers { permanent_uuid: "4c5611deb12845d982674dcfea33fd93" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 38277 } } peers { permanent_uuid: "2ab38de74582419e893b702c9dd7a051" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 37273 } }
I20260619 18:50:35.547204 20050 leader_election.cc:288] T 9658d380da634ccd98d12305a673e12b P bad6f9f3febb40e8b9161ba802d06542 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c5611deb12845d982674dcfea33fd93 (127.3.243.65:38277), 2ab38de74582419e893b702c9dd7a051 (127.3.243.66:37273)
I20260619 18:50:35.552798 20026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:35.547453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.66:37273 (local address 127.3.243.67:37655)
0619 18:50:35.547757 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:35.547781 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 18:50:35.547923 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:50:35.548321 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:50:35.548325 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:50:35.548349 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:50:35.548523 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:50:35.548534 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:50:35.549799 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:50:35.549804 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:50:35.551074 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:50:35.551084 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:50:35.552106 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:50:35.552112 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:50:35.552292 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:35.552323 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 18:50:35.552400 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 18:50:35.552526 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":23}
I20260619 18:50:35.552790 20041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:35.547577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.67:37655 (local address 127.3.243.66:37273)
0619 18:50:35.547760 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 18:50:35.547766 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:50:35.547857 (+ 91us) server_negotiation.cc:407] Connection header received
0619 18:50:35.548087 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:50:35.548093 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:50:35.548204 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:50:35.548302 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:50:35.548672 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:35.549677 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:35.551217 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:35.551969 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:35.552099 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:35.552192 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 18:50:35.552437 (+ 245us) server_negotiation.cc:299] Negotiation successful
0619 18:50:35.552559 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 18:50:35.553681 20039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:35.547680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.67:59833 (local address 127.3.243.65:38277)
0619 18:50:35.547869 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 18:50:35.547875 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:50:35.548483 (+ 608us) server_negotiation.cc:407] Connection header received
0619 18:50:35.548636 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:50:35.548641 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:50:35.548749 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:50:35.548842 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:50:35.549428 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:35.550201 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:35.551830 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:35.552650 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:35.552805 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:35.552873 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 18:50:35.553316 (+ 443us) server_negotiation.cc:299] Negotiation successful
0619 18:50:35.553436 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 18:50:35.553733 20051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:35.547560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.65:38277 (local address 127.3.243.67:59833)
0619 18:50:35.548369 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:35.548393 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 18:50:35.548504 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:50:35.548875 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:50:35.548879 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:50:35.548904 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:50:35.549253 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:50:35.549265 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:50:35.550346 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:50:35.550349 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:50:35.551679 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:50:35.551690 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:50:35.553008 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:50:35.553013 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:50:35.553170 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:35.553194 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 18:50:35.553271 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 18:50:35.553382 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":688,"mutex_wait_us":20,"thread_start_us":438,"threads_started":1}
I20260619 18:50:35.553718 19846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9658d380da634ccd98d12305a673e12b" candidate_uuid: "bad6f9f3febb40e8b9161ba802d06542" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ab38de74582419e893b702c9dd7a051" is_pre_election: true
I20260619 18:50:35.554268 19846 raft_consensus.cc:2454] T 9658d380da634ccd98d12305a673e12b P 2ab38de74582419e893b702c9dd7a051 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bad6f9f3febb40e8b9161ba802d06542 in term 0.
I20260619 18:50:35.554701 19712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9658d380da634ccd98d12305a673e12b" candidate_uuid: "bad6f9f3febb40e8b9161ba802d06542" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c5611deb12845d982674dcfea33fd93" is_pre_election: true
I20260619 18:50:35.554863 19915 leader_election.cc:302] T 9658d380da634ccd98d12305a673e12b P bad6f9f3febb40e8b9161ba802d06542 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ab38de74582419e893b702c9dd7a051, bad6f9f3febb40e8b9161ba802d06542; no voters:
I20260619 18:50:35.555224 20050 raft_consensus.cc:2781] T 9658d380da634ccd98d12305a673e12b P bad6f9f3febb40e8b9161ba802d06542 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:50:35.555207 19712 raft_consensus.cc:2454] T 9658d380da634ccd98d12305a673e12b P 4c5611deb12845d982674dcfea33fd93 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bad6f9f3febb40e8b9161ba802d06542 in term 0.
I20260619 18:50:35.555372 20050 raft_consensus.cc:482] T 9658d380da634ccd98d12305a673e12b P bad6f9f3febb40e8b9161ba802d06542 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:50:35.555475 20050 raft_consensus.cc:3037] T 9658d380da634ccd98d12305a673e12b P bad6f9f3febb40e8b9161ba802d06542 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:50:35.557741 20050 raft_consensus.cc:504] T 9658d380da634ccd98d12305a673e12b P bad6f9f3febb40e8b9161ba802d06542 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bad6f9f3febb40e8b9161ba802d06542" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40449 } } peers { permanent_uuid: "4c5611deb12845d982674dcfea33fd93" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 38277 } } peers { permanent_uuid: "2ab38de74582419e893b702c9dd7a051" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 37273 } }
I20260619 18:50:35.558280 20050 leader_election.cc:288] T 9658d380da634ccd98d12305a673e12b P bad6f9f3febb40e8b9161ba802d06542 [CANDIDATE]: Term 1 election: Requested vote from peers 4c5611deb12845d982674dcfea33fd93 (127.3.243.65:38277), 2ab38de74582419e893b702c9dd7a051 (127.3.243.66:37273)
I20260619 18:50:35.558619 19712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9658d380da634ccd98d12305a673e12b" candidate_uuid: "bad6f9f3febb40e8b9161ba802d06542" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c5611deb12845d982674dcfea33fd93"
I20260619 18:50:35.558772 19846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9658d380da634ccd98d12305a673e12b" candidate_uuid: "bad6f9f3febb40e8b9161ba802d06542" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ab38de74582419e893b702c9dd7a051"
I20260619 18:50:35.558856 19712 raft_consensus.cc:3037] T 9658d380da634ccd98d12305a673e12b P 4c5611deb12845d982674dcfea33fd93 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:50:35.558974 19846 raft_consensus.cc:3037] T 9658d380da634ccd98d12305a673e12b P 2ab38de74582419e893b702c9dd7a051 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:50:35.561890 19712 raft_consensus.cc:2454] T 9658d380da634ccd98d12305a673e12b P 4c5611deb12845d982674dcfea33fd93 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bad6f9f3febb40e8b9161ba802d06542 in term 1.
I20260619 18:50:35.561935 19846 raft_consensus.cc:2454] T 9658d380da634ccd98d12305a673e12b P 2ab38de74582419e893b702c9dd7a051 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bad6f9f3febb40e8b9161ba802d06542 in term 1.
I20260619 18:50:35.562273 19912 leader_election.cc:302] T 9658d380da634ccd98d12305a673e12b P bad6f9f3febb40e8b9161ba802d06542 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2ab38de74582419e893b702c9dd7a051, 4c5611deb12845d982674dcfea33fd93, bad6f9f3febb40e8b9161ba802d06542; no voters:
I20260619 18:50:35.562579 20050 raft_consensus.cc:2781] T 9658d380da634ccd98d12305a673e12b P bad6f9f3febb40e8b9161ba802d06542 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:50:35.562966 20050 raft_consensus.cc:686] T 9658d380da634ccd98d12305a673e12b P bad6f9f3febb40e8b9161ba802d06542 [term 1 LEADER]: Becoming Leader. State: Replica: bad6f9f3febb40e8b9161ba802d06542, State: Running, Role: LEADER
I20260619 18:50:35.563366 20050 consensus_queue.cc:237] T 9658d380da634ccd98d12305a673e12b P bad6f9f3febb40e8b9161ba802d06542 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bad6f9f3febb40e8b9161ba802d06542" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40449 } } peers { permanent_uuid: "4c5611deb12845d982674dcfea33fd93" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 38277 } } peers { permanent_uuid: "2ab38de74582419e893b702c9dd7a051" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 37273 } }
I20260619 18:50:35.567418 19568 catalog_manager.cc:5495] T 9658d380da634ccd98d12305a673e12b P bad6f9f3febb40e8b9161ba802d06542 reported cstate change: term changed from 0 to 1, leader changed from <none> to bad6f9f3febb40e8b9161ba802d06542 (127.3.243.67). New cstate: current_term: 1 leader_uuid: "bad6f9f3febb40e8b9161ba802d06542" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bad6f9f3febb40e8b9161ba802d06542" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40449 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c5611deb12845d982674dcfea33fd93" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 38277 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ab38de74582419e893b702c9dd7a051" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 37273 } health_report { overall_health: UNKNOWN } } }
I20260619 18:50:35.584213 20038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:35.578978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34602 (local address 127.3.243.67:40449)
0619 18:50:35.579151 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 18:50:35.579156 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:50:35.579200 (+ 44us) server_negotiation.cc:407] Connection header received
0619 18:50:35.579405 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:50:35.579408 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:50:35.579476 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:50:35.579594 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:50:35.579969 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:35.580702 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:35.582202 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:35.582615 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:35.582716 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:35.583190 (+ 474us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:50:35.583656 (+ 466us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:50:35.583788 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0619 18:50:35.583875 (+ 87us) server_negotiation.cc:299] Negotiation successful
0619 18:50:35.583949 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 18:50:35.591523 19846 raft_consensus.cc:1261] T 9658d380da634ccd98d12305a673e12b P 2ab38de74582419e893b702c9dd7a051 [term 1 FOLLOWER]: Refusing update from remote peer bad6f9f3febb40e8b9161ba802d06542: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:50:35.591539 19712 raft_consensus.cc:1261] T 9658d380da634ccd98d12305a673e12b P 4c5611deb12845d982674dcfea33fd93 [term 1 FOLLOWER]: Refusing update from remote peer bad6f9f3febb40e8b9161ba802d06542: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:50:35.592383 20052 consensus_queue.cc:1035] T 9658d380da634ccd98d12305a673e12b P bad6f9f3febb40e8b9161ba802d06542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ab38de74582419e893b702c9dd7a051" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 37273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:50:35.592698 20050 consensus_queue.cc:1035] T 9658d380da634ccd98d12305a673e12b P bad6f9f3febb40e8b9161ba802d06542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c5611deb12845d982674dcfea33fd93" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 38277 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:50:35.614732 19567 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40886:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:50:35.615022 19567 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40886:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:50:35.618577 19567 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5eb6468c42124bb8acf48ba3fbceac41: Sending DeleteTablet for 3 replicas of tablet 9658d380da634ccd98d12305a673e12b
I20260619 18:50:35.619591 19826 tablet_service.cc:1514] Processing DeleteTablet for tablet 9658d380da634ccd98d12305a673e12b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:50:35 UTC) from {username='kudu', principal='kudu/127.3.243.126@KRBTEST.COM'} at 127.0.0.1:33116
I20260619 18:50:35.619578 19959 tablet_service.cc:1514] Processing DeleteTablet for tablet 9658d380da634ccd98d12305a673e12b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:50:35 UTC) from {username='kudu', principal='kudu/127.3.243.126@KRBTEST.COM'} at 127.0.0.1:34600
I20260619 18:50:35.620220 19692 tablet_service.cc:1514] Processing DeleteTablet for tablet 9658d380da634ccd98d12305a673e12b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:50:35 UTC) from {username='kudu', principal='kudu/127.3.243.126@KRBTEST.COM'} at 127.0.0.1:35736
I20260619 18:50:35.620365 20065 tablet_replica.cc:330] T 9658d380da634ccd98d12305a673e12b P 2ab38de74582419e893b702c9dd7a051: stopping tablet replica
I20260619 18:50:35.620703 20065 raft_consensus.cc:2229] T 9658d380da634ccd98d12305a673e12b P 2ab38de74582419e893b702c9dd7a051 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:50:35.621083 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 19629
I20260619 18:50:35.621110 20065 raft_consensus.cc:2258] T 9658d380da634ccd98d12305a673e12b P 2ab38de74582419e893b702c9dd7a051 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:50:35.629736 20066 tablet_replica.cc:330] T 9658d380da634ccd98d12305a673e12b P bad6f9f3febb40e8b9161ba802d06542: stopping tablet replica
I20260619 18:50:35.629725 20065 ts_tablet_manager.cc:1905] T 9658d380da634ccd98d12305a673e12b P 2ab38de74582419e893b702c9dd7a051: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:50:35.630089 20066 raft_consensus.cc:2229] T 9658d380da634ccd98d12305a673e12b P bad6f9f3febb40e8b9161ba802d06542 [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:50:35.630700 20066 raft_consensus.cc:2258] T 9658d380da634ccd98d12305a673e12b P bad6f9f3febb40e8b9161ba802d06542 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:50:35.631410 20067 tablet_replica.cc:330] T 9658d380da634ccd98d12305a673e12b P 4c5611deb12845d982674dcfea33fd93: stopping tablet replica
I20260619 18:50:35.631845 20067 raft_consensus.cc:2229] T 9658d380da634ccd98d12305a673e12b P 4c5611deb12845d982674dcfea33fd93 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:50:35.632330 20067 raft_consensus.cc:2258] T 9658d380da634ccd98d12305a673e12b P 4c5611deb12845d982674dcfea33fd93 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:50:35.632653 20066 ts_tablet_manager.cc:1905] T 9658d380da634ccd98d12305a673e12b P bad6f9f3febb40e8b9161ba802d06542: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:50:35.634411 20067 ts_tablet_manager.cc:1905] T 9658d380da634ccd98d12305a673e12b P 4c5611deb12845d982674dcfea33fd93: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:50:35.637634 20065 ts_tablet_manager.cc:1918] T 9658d380da634ccd98d12305a673e12b P 2ab38de74582419e893b702c9dd7a051: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:50:35.637779 20065 log.cc:1196] T 9658d380da634ccd98d12305a673e12b P 2ab38de74582419e893b702c9dd7a051: Deleting WAL directory at /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-1/wal/wals/9658d380da634ccd98d12305a673e12b
I20260619 18:50:35.638307 20065 ts_tablet_manager.cc:1939] T 9658d380da634ccd98d12305a673e12b P 2ab38de74582419e893b702c9dd7a051: Deleting consensus metadata
I20260619 18:50:35.639775 20066 ts_tablet_manager.cc:1918] T 9658d380da634ccd98d12305a673e12b P bad6f9f3febb40e8b9161ba802d06542: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:50:35.639931 20066 log.cc:1196] T 9658d380da634ccd98d12305a673e12b P bad6f9f3febb40e8b9161ba802d06542: Deleting WAL directory at /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-2/wal/wals/9658d380da634ccd98d12305a673e12b
I20260619 18:50:35.639953 19555 catalog_manager.cc:4841] TS 2ab38de74582419e893b702c9dd7a051 (127.3.243.66:37273): tablet 9658d380da634ccd98d12305a673e12b (table test-table [id=436e6219f0d749769602763645dddbbd]) successfully deleted
I20260619 18:50:35.640481 20066 ts_tablet_manager.cc:1939] T 9658d380da634ccd98d12305a673e12b P bad6f9f3febb40e8b9161ba802d06542: Deleting consensus metadata
I20260619 18:50:35.642177 19554 catalog_manager.cc:4841] TS bad6f9f3febb40e8b9161ba802d06542 (127.3.243.67:40449): tablet 9658d380da634ccd98d12305a673e12b (table test-table [id=436e6219f0d749769602763645dddbbd]) successfully deleted
I20260619 18:50:35.643323 20067 ts_tablet_manager.cc:1918] T 9658d380da634ccd98d12305a673e12b P 4c5611deb12845d982674dcfea33fd93: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:50:35.643435 20067 log.cc:1196] T 9658d380da634ccd98d12305a673e12b P 4c5611deb12845d982674dcfea33fd93: Deleting WAL directory at /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-0/wal/wals/9658d380da634ccd98d12305a673e12b
I20260619 18:50:35.643823 20067 ts_tablet_manager.cc:1939] T 9658d380da634ccd98d12305a673e12b P 4c5611deb12845d982674dcfea33fd93: Deleting consensus metadata
I20260619 18:50:35.645147 19552 catalog_manager.cc:4841] TS 4c5611deb12845d982674dcfea33fd93 (127.3.243.65:38277): tablet 9658d380da634ccd98d12305a673e12b (table test-table [id=436e6219f0d749769602763645dddbbd]) successfully deleted
Jun 19 18:50:35 dist-test-slave-m9wg krb5kdc[19523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895033, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.65@KRBTEST.COM
I20260619 18:50:35.657864 20039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:35.622359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35744 (local address 127.3.243.65:38277)
0619 18:50:35.622531 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 18:50:35.622538 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:50:35.633378 (+ 10840us) server_negotiation.cc:407] Connection header received
0619 18:50:35.638120 (+ 4742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:50:35.638126 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:50:35.638205 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:50:35.638279 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:50:35.640548 (+ 2269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:35.641728 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:35.646296 (+ 4568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:35.646695 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:35.646786 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:35.651621 (+ 4835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:50:35.651635 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:50:35.651653 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:50:35.651693 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:50:35.655169 (+ 3476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:35.655802 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:35.655812 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:35.655816 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:35.655976 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:35.656357 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:35.656362 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:35.656366 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:35.656806 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:50:35.657106 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0619 18:50:35.657282 (+ 176us) server_negotiation.cc:299] Negotiation successful
0619 18:50:35.657496 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":55}
I20260619 18:50:35.658470 19752 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:50:35.744832 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 19629
I20260619 18:50:35.759014 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 19763
Jun 19 18:50:35 dist-test-slave-m9wg krb5kdc[19523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895033, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.66@KRBTEST.COM
I20260619 18:50:35.774489 20041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:35.759922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33130 (local address 127.3.243.66:37273)
0619 18:50:35.760109 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 18:50:35.760115 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:50:35.760167 (+ 52us) server_negotiation.cc:407] Connection header received
0619 18:50:35.760230 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:50:35.760234 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:50:35.760306 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:50:35.760389 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:50:35.761612 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:35.762353 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:35.763645 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:35.764039 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:35.764268 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:35.768756 (+ 4488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:50:35.768770 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:50:35.768784 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:50:35.768824 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:50:35.771387 (+ 2563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:35.772260 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:35.772270 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:35.772274 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:35.772425 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:35.772864 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:35.772870 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:35.772873 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:35.773399 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:50:35.773708 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0619 18:50:35.773895 (+ 187us) server_negotiation.cc:299] Negotiation successful
0619 18:50:35.774125 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 18:50:35.775168 19886 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:50:35.921568 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 19763
I20260619 18:50:35.935573 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 19896
Jun 19 18:50:35 dist-test-slave-m9wg krb5kdc[19523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895033, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.67@KRBTEST.COM
I20260619 18:50:35.951129 20038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:35.936324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34618 (local address 127.3.243.67:40449)
0619 18:50:35.936500 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 18:50:35.936506 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:50:35.936625 (+ 119us) server_negotiation.cc:407] Connection header received
0619 18:50:35.936831 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:50:35.936835 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:50:35.936924 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:50:35.937013 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:50:35.938332 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:35.939153 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:35.940567 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:35.941006 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:35.941120 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:35.945316 (+ 4196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:50:35.945328 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:50:35.945344 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:50:35.945378 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:50:35.948500 (+ 3122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:35.949137 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:35.949144 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:35.949146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:35.949303 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:35.949669 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:35.949673 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:35.949675 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:35.950021 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:50:35.950317 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0619 18:50:35.950604 (+ 287us) server_negotiation.cc:299] Negotiation successful
0619 18:50:35.950816 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 18:50:35.951802 20019 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:50:36.072751 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 19896
I20260619 18:50:36.088402 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 19536
I20260619 18:50:36.089435 19598 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:50:36.195839 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 19536
I20260619 18:50:56.213374 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.126:42879
--webserver_interface=127.3.243.126
--webserver_port=46413
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab
--principal=kudu/127.3.243.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:45379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.126:42879
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:50:56.520272 20073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:50:56.520530 20073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:50:56.520629 20073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:50:56.527518 20073 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 18:50:56.527611 20073 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 18:50:56.527647 20073 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 18:50:56.527696 20073 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:50:56.527745 20073 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:50:56.527808 20073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:50:56.527850 20073 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:50:56.527900 20073 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:50:56.537447 20073 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:45379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.3.243.126:42879
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab
--principal=kudu/127.3.243.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.126:42879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.243.126
--webserver_port=46413
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:50:56.538590 20073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:50:56.539786 20073 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:50:56.560429 20082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:50:56.560510 20079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:50:56.560509 20080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:50:56.561362 20073 server_base.cc:1034] running on GCE node
I20260619 18:50:56.562202 20073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:50:56.563771 20073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:50:56.564981 20073 hybrid_clock.cc:648] HybridClock initialized: now 1781895056564913 us; error 41 us; skew 500 ppm
I20260619 18:50:56.565357 20073 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:50:56 dist-test-slave-m9wg krb5kdc[19523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895056, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:50:56.568749 20073 init.cc:377] Logged in from keytab as kudu/127.3.243.126@KRBTEST.COM (short username kudu)
I20260619 18:50:56.570263 20073 webserver.cc:460] Webserver started at http://127.3.243.126:46413/ using document root <none> and password file <none>
I20260619 18:50:56.571007 20073 fs_manager.cc:359] Metadata directory not provided
I20260619 18:50:56.571125 20073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:50:56.576867 20073 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260619 18:50:56.579630 20089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:50:56.580953 20073 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 18:50:56.581118 20073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/master-0/wal
uuid: "5eb6468c42124bb8acf48ba3fbceac41"
format_stamp: "Formatted at 2026-06-19 18:50:33 on dist-test-slave-m9wg"
server_key: "5bee5c99c932f10c288f5b8278cc5e63"
server_key_iv: "9b5ff271c13b1f6b55a7ae9a06a2e25f"
server_key_version: "encryptionkey@0"
I20260619 18:50:56.582103 20073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:50:56.595827 20073 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:50:56.599851 20073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:50:56.600131 20073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:50:56.616181 20073 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.126:42879
I20260619 18:50:56.616223 20140 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.126:42879 every 8 connection(s)
I20260619 18:50:56.617632 20073 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
I20260619 18:50:56.624001 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 20073
I20260619 18:50:56.624251 20141 sys_catalog.cc:263] Verifying existing consensus state
I20260619 18:50:56.624825 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.65:38277
--local_ip_for_outbound_sockets=127.3.243.65
--tserver_master_addrs=127.3.243.126:42879
--webserver_port=34305
--webserver_interface=127.3.243.65
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.65.keytab
--principal=kudu/127.3.243.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:45379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 18:50:56.627401 20141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5eb6468c42124bb8acf48ba3fbceac41: Bootstrap starting.
I20260619 18:50:56.648967 20141 log.cc:824] T 00000000000000000000000000000000 P 5eb6468c42124bb8acf48ba3fbceac41: Log is configured to *not* fsync() on all Append() calls
I20260619 18:50:56.660771 20141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5eb6468c42124bb8acf48ba3fbceac41: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 18:50:56.661235 20141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5eb6468c42124bb8acf48ba3fbceac41: Bootstrap complete.
I20260619 18:50:56.669785 20141 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5eb6468c42124bb8acf48ba3fbceac41 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5eb6468c42124bb8acf48ba3fbceac41" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 42879 } }
I20260619 18:50:56.670586 20141 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5eb6468c42124bb8acf48ba3fbceac41 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5eb6468c42124bb8acf48ba3fbceac41, State: Initialized, Role: FOLLOWER
I20260619 18:50:56.671180 20141 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5eb6468c42124bb8acf48ba3fbceac41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5eb6468c42124bb8acf48ba3fbceac41" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 42879 } }
I20260619 18:50:56.671398 20141 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5eb6468c42124bb8acf48ba3fbceac41 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:50:56.671504 20141 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5eb6468c42124bb8acf48ba3fbceac41 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:50:56.671658 20141 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5eb6468c42124bb8acf48ba3fbceac41 [term 1 FOLLOWER]: Advancing to term 2
I20260619 18:50:56.675215 20141 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5eb6468c42124bb8acf48ba3fbceac41 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5eb6468c42124bb8acf48ba3fbceac41" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 42879 } }
I20260619 18:50:56.675680 20141 leader_election.cc:302] T 00000000000000000000000000000000 P 5eb6468c42124bb8acf48ba3fbceac41 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5eb6468c42124bb8acf48ba3fbceac41; no voters:
I20260619 18:50:56.676275 20141 leader_election.cc:288] T 00000000000000000000000000000000 P 5eb6468c42124bb8acf48ba3fbceac41 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 18:50:56.676455 20145 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5eb6468c42124bb8acf48ba3fbceac41 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 18:50:56.677445 20145 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5eb6468c42124bb8acf48ba3fbceac41 [term 2 LEADER]: Becoming Leader. State: Replica: 5eb6468c42124bb8acf48ba3fbceac41, State: Running, Role: LEADER
I20260619 18:50:56.678038 20145 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5eb6468c42124bb8acf48ba3fbceac41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5eb6468c42124bb8acf48ba3fbceac41" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 42879 } }
I20260619 18:50:56.678205 20141 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5eb6468c42124bb8acf48ba3fbceac41 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:50:56.680856 20147 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5eb6468c42124bb8acf48ba3fbceac41 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5eb6468c42124bb8acf48ba3fbceac41. Latest consensus state: current_term: 2 leader_uuid: "5eb6468c42124bb8acf48ba3fbceac41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5eb6468c42124bb8acf48ba3fbceac41" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 42879 } } }
I20260619 18:50:56.681177 20147 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5eb6468c42124bb8acf48ba3fbceac41 [sys.catalog]: This master's current role is: LEADER
I20260619 18:50:56.681694 20146 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5eb6468c42124bb8acf48ba3fbceac41 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "5eb6468c42124bb8acf48ba3fbceac41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5eb6468c42124bb8acf48ba3fbceac41" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 42879 } } }
I20260619 18:50:56.681985 20146 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5eb6468c42124bb8acf48ba3fbceac41 [sys.catalog]: This master's current role is: LEADER
I20260619 18:50:56.691442 20152 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:50:56.698880 20152 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:50:56.700268 20152 catalog_manager.cc:1259] Loaded cluster ID: c15cbeedde704a27b2d2b3bbd371d776
I20260619 18:50:56.700356 20152 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:50:56.704120 20152 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:50:56.705569 20152 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5eb6468c42124bb8acf48ba3fbceac41: Loaded TSK: 0
I20260619 18:50:56.736726 20152 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5eb6468c42124bb8acf48ba3fbceac41: Generated new TSK 1
I20260619 18:50:56.737097 20152 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 18:50:56.962245 20143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:50:56.962483 20143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:50:56.962606 20143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:50:56.969506 20143 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:50:56.969604 20143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:50:56.969743 20143 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.65
I20260619 18:50:56.979017 20143 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:45379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.65.keytab
--principal=kudu/127.3.243.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.65:38277
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.243.65
--webserver_port=34305
--tserver_master_addrs=127.3.243.126:42879
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.65
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:50:56.980213 20143 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:50:56.981453 20143 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:50:56.995656 20169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:50:56.995841 20171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:50:56.996352 20168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:50:56.997488 20143 server_base.cc:1034] running on GCE node
I20260619 18:50:56.998080 20143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:50:56.998989 20143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:50:57.000221 20143 hybrid_clock.cc:648] HybridClock initialized: now 1781895057000166 us; error 46 us; skew 500 ppm
I20260619 18:50:57.000511 20143 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:50:56 dist-test-slave-m9wg krb5kdc[19523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895056, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:50:57.003464 20143 init.cc:377] Logged in from keytab as kudu/127.3.243.65@KRBTEST.COM (short username kudu)
I20260619 18:50:57.004904 20143 webserver.cc:460] Webserver started at http://127.3.243.65:34305/ using document root <none> and password file <none>
I20260619 18:50:57.005548 20143 fs_manager.cc:359] Metadata directory not provided
I20260619 18:50:57.005676 20143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:50:57.010895 20143 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 18:50:57.013563 20178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:50:57.014923 20143 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:50:57.015084 20143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-0/wal
uuid: "4c5611deb12845d982674dcfea33fd93"
format_stamp: "Formatted at 2026-06-19 18:50:34 on dist-test-slave-m9wg"
server_key: "8949a1fc82148059d2455b1dfd58e8bb"
server_key_iv: "eb1da66fec32d889e1dc8f419c86652b"
server_key_version: "encryptionkey@0"
I20260619 18:50:57.015974 20143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:50:57.053419 20143 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:50:57.057224 20143 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:50:57.057471 20143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:50:57.058432 20143 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:50:57.059823 20143 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:50:57.059923 20143 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:50:57.060027 20143 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:50:57.060101 20143 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:50:57.089900 20143 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.65:38277
I20260619 18:50:57.089960 20290 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.65:38277 every 8 connection(s)
I20260619 18:50:57.091277 20143 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-0/data/info.pb
I20260619 18:50:57.099620 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 20143
I20260619 18:50:57.100414 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.66:37273
--local_ip_for_outbound_sockets=127.3.243.66
--tserver_master_addrs=127.3.243.126:42879
--webserver_port=36089
--webserver_interface=127.3.243.66
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.66.keytab
--principal=kudu/127.3.243.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:45379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 18:50:57 dist-test-slave-m9wg krb5kdc[19523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895056, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.65@KRBTEST.COM for kudu/127.3.243.126@KRBTEST.COM
I20260619 18:50:57.114074 20293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:57.094095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.65:60709 (local address 127.3.243.126:42879)
0619 18:50:57.094651 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0619 18:50:57.094662 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 18:50:57.095336 (+ 674us) server_negotiation.cc:407] Connection header received
0619 18:50:57.097486 (+ 2150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:50:57.097532 (+ 46us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:50:57.097873 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:50:57.098162 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:50:57.099570 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:57.101445 (+ 1875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:57.102950 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:57.103865 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:57.104037 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:57.107367 (+ 3330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:50:57.107378 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:50:57.107396 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:50:57.107439 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:50:57.110116 (+ 2677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:57.110925 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:57.110931 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:57.110937 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:57.111115 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:57.111515 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:57.111520 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:57.111523 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:57.111950 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:50:57.112102 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 18:50:57.112906 (+ 804us) server_negotiation.cc:299] Negotiation successful
0619 18:50:57.113206 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":208,"threads_started":1}
I20260619 18:50:57.114356 20294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:57.094381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:42879 (local address 127.3.243.65:60709)
0619 18:50:57.095185 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:57.095234 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 18:50:57.096962 (+ 1728us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:50:57.098288 (+ 1326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:50:57.098295 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:50:57.098797 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:50:57.099373 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:50:57.099391 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:50:57.101614 (+ 2223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:50:57.101618 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:50:57.102813 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:50:57.102829 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:50:57.104078 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:50:57.104083 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:50:57.104302 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:57.104930 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:50:57.104952 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:50:57.107092 (+ 2140us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:50:57.110250 (+ 3158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:50:57.110258 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:50:57.110272 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:50:57.110805 (+ 533us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:50:57.111205 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:50:57.111209 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:50:57.111212 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:50:57.111420 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:50:57.112092 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:50:57.112099 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:50:57.112470 (+ 371us) client_negotiation.cc:769] Sending connection context
0619 18:50:57.112753 (+ 283us) client_negotiation.cc:240] Negotiation successful
0619 18:50:57.113234 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":304,"threads_started":1}
I20260619 18:50:57.116991 20291 heartbeater.cc:347] Connected to a master server at 127.3.243.126:42879
I20260619 18:50:57.117317 20291 heartbeater.cc:464] Registering TS with master...
I20260619 18:50:57.118184 20291 heartbeater.cc:511] Master 127.3.243.126:42879 requested a full tablet report, sending...
I20260619 18:50:57.120487 20106 ts_manager.cc:194] Registered new tserver with Master: 4c5611deb12845d982674dcfea33fd93 (127.3.243.65:38277)
I20260619 18:50:57.121901 20106 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.65@KRBTEST.COM'} at 127.3.243.65:60709
W20260619 18:50:57.409646 20295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:50:57.409888 20295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:50:57.410015 20295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:50:57.416926 20295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:50:57.417014 20295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:50:57.417166 20295 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.66
I20260619 18:50:57.426435 20295 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:45379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.66.keytab
--principal=kudu/127.3.243.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.66:37273
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.243.66
--webserver_port=36089
--tserver_master_addrs=127.3.243.126:42879
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.66
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:50:57.427613 20295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:50:57.428881 20295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:50:57.440913 20300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:50:57.440951 20301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:50:57.441241 20303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:50:57.441969 20295 server_base.cc:1034] running on GCE node
I20260619 18:50:57.442519 20295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:50:57.443359 20295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:50:57.444550 20295 hybrid_clock.cc:648] HybridClock initialized: now 1781895057444498 us; error 44 us; skew 500 ppm
I20260619 18:50:57.444836 20295 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:50:57 dist-test-slave-m9wg krb5kdc[19523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895057, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:50:57.452071 20295 init.cc:377] Logged in from keytab as kudu/127.3.243.66@KRBTEST.COM (short username kudu)
I20260619 18:50:57.453444 20295 webserver.cc:460] Webserver started at http://127.3.243.66:36089/ using document root <none> and password file <none>
I20260619 18:50:57.454088 20295 fs_manager.cc:359] Metadata directory not provided
I20260619 18:50:57.454211 20295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:50:57.459307 20295 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 18:50:57.461974 20310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:50:57.463156 20295 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 18:50:57.463289 20295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-1/wal
uuid: "2ab38de74582419e893b702c9dd7a051"
format_stamp: "Formatted at 2026-06-19 18:50:34 on dist-test-slave-m9wg"
server_key: "2e62b509e7add8e89f6a640ab71367ab"
server_key_iv: "4bdd0a384976c71967d5e44491c4d209"
server_key_version: "encryptionkey@0"
I20260619 18:50:57.464193 20295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:50:57.483085 20295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:50:57.486626 20295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:50:57.486864 20295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:50:57.487746 20295 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:50:57.488999 20295 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:50:57.489099 20295 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:50:57.489202 20295 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:50:57.489269 20295 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:50:57.519061 20295 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.66:37273
I20260619 18:50:57.519146 20422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.66:37273 every 8 connection(s)
I20260619 18:50:57.520668 20295 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-1/data/info.pb
I20260619 18:50:57.522953 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 20295
I20260619 18:50:57.523694 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.67:40449
--local_ip_for_outbound_sockets=127.3.243.67
--tserver_master_addrs=127.3.243.126:42879
--webserver_port=45589
--webserver_interface=127.3.243.67
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.67.keytab
--principal=kudu/127.3.243.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:45379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 18:50:57 dist-test-slave-m9wg krb5kdc[19523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895057, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.66@KRBTEST.COM for kudu/127.3.243.126@KRBTEST.COM
I20260619 18:50:57.540359 20293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:57.523209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.66:39429 (local address 127.3.243.126:42879)
0619 18:50:57.523378 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 18:50:57.523382 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:50:57.524740 (+ 1358us) server_negotiation.cc:407] Connection header received
0619 18:50:57.526542 (+ 1802us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:50:57.526548 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:50:57.526628 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:50:57.526710 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:50:57.528194 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:57.528880 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:57.530660 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:57.531080 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:57.531180 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:57.534979 (+ 3799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:50:57.534984 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:50:57.534988 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:50:57.535020 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:50:57.537341 (+ 2321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:57.538130 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:57.538134 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:57.538136 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:57.538266 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:57.538665 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:57.538668 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:57.538670 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:57.538946 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:50:57.539059 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 18:50:57.539887 (+ 828us) server_negotiation.cc:299] Negotiation successful
0619 18:50:57.540071 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 18:50:57.541801 20425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:57.523865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:42879 (local address 127.3.243.66:39429)
0619 18:50:57.524583 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:57.524626 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 18:50:57.526270 (+ 1644us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:50:57.526966 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:50:57.526975 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:50:57.527449 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:50:57.527999 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:50:57.528017 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:50:57.529025 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:50:57.529029 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:50:57.530505 (+ 1476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:50:57.530520 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:50:57.531305 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:50:57.531311 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:50:57.531563 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:57.532250 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:50:57.532282 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:50:57.534743 (+ 2461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:50:57.537462 (+ 2719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:50:57.537471 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:50:57.537485 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:50:57.538002 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:50:57.538362 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:50:57.538366 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:50:57.538370 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:50:57.538576 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:50:57.539054 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:50:57.539062 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:50:57.539445 (+ 383us) client_negotiation.cc:769] Sending connection context
0619 18:50:57.539810 (+ 365us) client_negotiation.cc:240] Negotiation successful
0619 18:50:57.540340 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":297,"threads_started":1}
I20260619 18:50:57.543294 20423 heartbeater.cc:347] Connected to a master server at 127.3.243.126:42879
I20260619 18:50:57.543563 20423 heartbeater.cc:464] Registering TS with master...
I20260619 18:50:57.544288 20423 heartbeater.cc:511] Master 127.3.243.126:42879 requested a full tablet report, sending...
I20260619 18:50:57.545639 20106 ts_manager.cc:194] Registered new tserver with Master: 2ab38de74582419e893b702c9dd7a051 (127.3.243.66:37273)
I20260619 18:50:57.546392 20106 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.66@KRBTEST.COM'} at 127.3.243.66:39429
W20260619 18:50:57.830772 20426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:50:57.831048 20426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:50:57.831183 20426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:50:57.838217 20426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:50:57.838335 20426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:50:57.838478 20426 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.67
I20260619 18:50:57.848374 20426 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:45379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.67.keytab
--principal=kudu/127.3.243.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.67:40449
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.243.67
--webserver_port=45589
--tserver_master_addrs=127.3.243.126:42879
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.67
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:50:57.849614 20426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:50:57.850875 20426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:50:57.863301 20431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:50:57.863276 20434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:50:57.863240 20432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:50:57.864253 20426 server_base.cc:1034] running on GCE node
I20260619 18:50:57.864818 20426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:50:57.865721 20426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:50:57.866899 20426 hybrid_clock.cc:648] HybridClock initialized: now 1781895057866854 us; error 43 us; skew 500 ppm
I20260619 18:50:57.867192 20426 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:50:57 dist-test-slave-m9wg krb5kdc[19523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895057, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:50:57.870277 20426 init.cc:377] Logged in from keytab as kudu/127.3.243.67@KRBTEST.COM (short username kudu)
I20260619 18:50:57.871670 20426 webserver.cc:460] Webserver started at http://127.3.243.67:45589/ using document root <none> and password file <none>
I20260619 18:50:57.872365 20426 fs_manager.cc:359] Metadata directory not provided
I20260619 18:50:57.872493 20426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:50:57.877862 20426 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 18:50:57.880560 20441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:50:57.881858 20426 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:50:57.882004 20426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-2/wal
uuid: "bad6f9f3febb40e8b9161ba802d06542"
format_stamp: "Formatted at 2026-06-19 18:50:35 on dist-test-slave-m9wg"
server_key: "3cedacdc2802ab3a8ff6cf01cef22bf4"
server_key_iv: "d46435461ac1345cae481d52c8f3b7e8"
server_key_version: "encryptionkey@0"
I20260619 18:50:57.882933 20426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:50:57.907989 20426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:50:57.911898 20426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:50:57.912166 20426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:50:57.913105 20426 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:50:57.914355 20426 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:50:57.914449 20426 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:50:57.914563 20426 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:50:57.914633 20426 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:50:57.943887 20426 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.67:40449
I20260619 18:50:57.943945 20553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.67:40449 every 8 connection(s)
I20260619 18:50:57.945348 20426 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-2/data/info.pb
I20260619 18:50:57.947021 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 20426
Jun 19 18:50:57 dist-test-slave-m9wg krb5kdc[19523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895057, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.67@KRBTEST.COM for kudu/127.3.243.126@KRBTEST.COM
I20260619 18:50:57.962284 20293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:57.947957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45214 (local address 127.3.243.126:42879)
0619 18:50:57.948307 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0619 18:50:57.948313 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:50:57.948593 (+ 280us) server_negotiation.cc:407] Connection header received
0619 18:50:57.948809 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:50:57.948814 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:50:57.948884 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:50:57.948964 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:50:57.950588 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:57.951299 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:57.952712 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:57.953254 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:57.953390 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:57.956582 (+ 3192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:50:57.956587 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:50:57.956590 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:50:57.956629 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:50:57.959398 (+ 2769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:57.960068 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:57.960071 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:57.960077 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:57.960291 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:57.960698 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:57.960703 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:57.960707 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:57.961030 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:50:57.961303 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0619 18:50:57.961640 (+ 337us) server_negotiation.cc:299] Negotiation successful
0619 18:50:57.961842 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":212}
I20260619 18:50:57.965909 20557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:57.947999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.67:53223 (local address 127.3.243.126:42879)
0619 18:50:57.949210 (+ 1211us) server_negotiation.cc:206] Beginning negotiation
0619 18:50:57.949214 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:50:57.949589 (+ 375us) server_negotiation.cc:407] Connection header received
0619 18:50:57.951382 (+ 1793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:50:57.951387 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:50:57.951462 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:50:57.951549 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:50:57.953059 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:57.953929 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:57.956193 (+ 2264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:57.956603 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:57.956723 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:57.959993 (+ 3270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:50:57.960000 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:50:57.960004 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:50:57.960040 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:50:57.963123 (+ 3083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:57.963885 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:57.963888 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:57.963890 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:57.963989 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:57.964477 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:57.964481 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:57.964484 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:57.964820 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:50:57.964930 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 18:50:57.965480 (+ 550us) server_negotiation.cc:299] Negotiation successful
0619 18:50:57.965653 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":1115,"thread_start_us":670,"threads_started":1}
I20260619 18:50:57.966797 20558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:57.948623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:42879 (local address 127.3.243.67:53223)
0619 18:50:57.949431 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:57.949480 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 18:50:57.951129 (+ 1649us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:50:57.951804 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:50:57.951812 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:50:57.952337 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:50:57.952885 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:50:57.952902 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:50:57.954085 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:50:57.954091 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:50:57.956011 (+ 1920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:50:57.956026 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:50:57.956837 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:50:57.956845 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:50:57.957081 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:57.957672 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:50:57.957692 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:50:57.959793 (+ 2101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:50:57.963249 (+ 3456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:50:57.963255 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:50:57.963265 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:50:57.963770 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:50:57.964103 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:50:57.964108 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:50:57.964111 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:50:57.964377 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:50:57.964930 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:50:57.964935 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:50:57.965204 (+ 269us) client_negotiation.cc:769] Sending connection context
0619 18:50:57.965417 (+ 213us) client_negotiation.cc:240] Negotiation successful
0619 18:50:57.965745 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":315,"threads_started":1}
I20260619 18:50:57.968008 20554 heartbeater.cc:347] Connected to a master server at 127.3.243.126:42879
I20260619 18:50:57.968317 20554 heartbeater.cc:464] Registering TS with master...
I20260619 18:50:57.968946 20554 heartbeater.cc:511] Master 127.3.243.126:42879 requested a full tablet report, sending...
I20260619 18:50:57.970191 20106 ts_manager.cc:194] Registered new tserver with Master: bad6f9f3febb40e8b9161ba802d06542 (127.3.243.67:40449)
I20260619 18:50:57.970935 20106 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.67@KRBTEST.COM'} at 127.3.243.67:53223
I20260619 18:50:57.976581 4045 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 18:50:57.982141 20557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:57.977388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45216 (local address 127.3.243.126:42879)
0619 18:50:57.977555 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 18:50:57.977560 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:50:57.977944 (+ 384us) server_negotiation.cc:407] Connection header received
0619 18:50:57.978091 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:50:57.978095 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:50:57.978215 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:50:57.978316 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:50:57.978665 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:57.979363 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:57.980686 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:57.981074 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:57.981158 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:57.981499 (+ 341us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:50:57.981694 (+ 195us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0619 18:50:57.981923 (+ 229us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45216: token expired
Metrics: {"server-negotiator.queue_time_us":49}
W20260619 18:50:57.982249 20557 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45216: token expired
W20260619 18:50:57.982750 20030 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.3.243.126:42879)
I20260619 18:50:57.994436 20557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:57.983124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45222 (local address 127.3.243.126:42879)
0619 18:50:57.983252 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 18:50:57.983255 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:50:57.983307 (+ 52us) server_negotiation.cc:407] Connection header received
0619 18:50:57.983409 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:50:57.983412 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:50:57.983457 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:50:57.983521 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:50:57.984818 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:57.985564 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:57.986980 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:57.987373 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:57.987465 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:57.989855 (+ 2390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:50:57.989861 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:50:57.989866 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:50:57.989902 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:50:57.992103 (+ 2201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:57.992834 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:57.992837 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:57.992839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:57.992942 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:57.993293 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:57.993297 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:57.993299 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:57.993596 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:50:57.993797 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0619 18:50:57.993993 (+ 196us) server_negotiation.cc:299] Negotiation successful
0619 18:50:57.994160 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 18:50:57.997447 20106 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45222:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:50:58.004607 20106 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:50:58.039028 20568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:58.015659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.67:40449 (local address 127.0.0.1:55544)
0619 18:50:58.027614 (+ 11955us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:58.027660 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 18:50:58.027881 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:50:58.031045 (+ 3164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:50:58.031053 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:50:58.031089 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:50:58.031771 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:50:58.031792 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:50:58.033575 (+ 1783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:50:58.033580 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:50:58.036313 (+ 2733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:50:58.036328 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:50:58.038144 (+ 1816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:50:58.038149 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:50:58.038298 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:58.038334 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 18:50:58.038464 (+ 130us) client_negotiation.cc:240] Negotiation successful
0619 18:50:58.038657 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11673,"thread_start_us":19206,"threads_started":1}
I20260619 18:50:58.039011 20566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:58.015939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55544 (local address 127.3.243.67:40449)
0619 18:50:58.030046 (+ 14107us) server_negotiation.cc:206] Beginning negotiation
0619 18:50:58.030056 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:50:58.030092 (+ 36us) server_negotiation.cc:407] Connection header received
0619 18:50:58.030205 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:50:58.030214 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:50:58.030488 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:50:58.030691 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:50:58.032014 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:58.033422 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:58.036504 (+ 3082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:58.037674 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:58.037803 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:58.038018 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0619 18:50:58.038518 (+ 500us) server_negotiation.cc:299] Negotiation successful
0619 18:50:58.038680 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13917,"thread_start_us":15623,"threads_started":1}
I20260619 18:50:58.041404 20565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:58.015773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.65:38277 (local address 127.0.0.1:49464)
0619 18:50:58.033246 (+ 17473us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:58.033273 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0619 18:50:58.033408 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:50:58.034251 (+ 843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:50:58.034257 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:50:58.034283 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:50:58.034684 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:50:58.034697 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:50:58.036548 (+ 1851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:50:58.036553 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:50:58.038562 (+ 2009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:50:58.038573 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:50:58.040652 (+ 2079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:50:58.040661 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:50:58.040825 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:58.040856 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 18:50:58.040945 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 18:50:58.041069 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17342,"thread_start_us":17733,"threads_started":1}
I20260619 18:50:58.042131 20567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:58.016290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49464 (local address 127.3.243.65:38277)
0619 18:50:58.026984 (+ 10694us) server_negotiation.cc:206] Beginning negotiation
0619 18:50:58.026994 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:50:58.033399 (+ 6405us) server_negotiation.cc:407] Connection header received
0619 18:50:58.033543 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:50:58.033551 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:50:58.033810 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:50:58.034005 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:50:58.034881 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:58.036406 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:58.038736 (+ 2330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:58.040225 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:58.040360 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:58.040546 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0619 18:50:58.041666 (+ 1120us) server_negotiation.cc:299] Negotiation successful
0619 18:50:58.041833 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10497,"thread_start_us":10357,"threads_started":1}
I20260619 18:50:58.042807 20564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:58.016277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.66:37273 (local address 127.0.0.1:60004)
0619 18:50:58.035137 (+ 18860us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:58.035151 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 18:50:58.035259 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:50:58.035739 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:50:58.035744 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:50:58.035766 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:50:58.036103 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:50:58.036112 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:50:58.038445 (+ 2333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:50:58.038451 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:50:58.040419 (+ 1968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:50:58.040433 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:50:58.042123 (+ 1690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:50:58.042129 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:50:58.042289 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:58.042319 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 18:50:58.042386 (+ 67us) client_negotiation.cc:240] Negotiation successful
0619 18:50:58.042504 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18745,"thread_start_us":10909,"threads_started":1}
I20260619 18:50:58.043550 20569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:58.016451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60004 (local address 127.3.243.66:37273)
0619 18:50:58.032134 (+ 15683us) server_negotiation.cc:206] Beginning negotiation
0619 18:50:58.032142 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:50:58.035251 (+ 3109us) server_negotiation.cc:407] Connection header received
0619 18:50:58.035417 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:50:58.035423 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:50:58.035575 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:50:58.035732 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:50:58.036313 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:58.038267 (+ 1954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:58.040606 (+ 2339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:58.041923 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:58.042032 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:58.043150 (+ 1118us) server_negotiation.cc:1035] Waiting for connection context
0619 18:50:58.043217 (+ 67us) server_negotiation.cc:299] Negotiation successful
0619 18:50:58.043335 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15513,"thread_start_us":15830,"threads_started":1}
I20260619 18:50:58.043491 20489 tablet_service.cc:1467] Processing CreateTablet for tablet 100545edd4ac4748b63f1f782249c369 (DEFAULT_TABLE table=test-table [id=d747f90d67ec418385149027cdbd040d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:50:58.045096 20489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 100545edd4ac4748b63f1f782249c369. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:50:58.045833 20226 tablet_service.cc:1467] Processing CreateTablet for tablet 100545edd4ac4748b63f1f782249c369 (DEFAULT_TABLE table=test-table [id=d747f90d67ec418385149027cdbd040d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:50:58.046556 20358 tablet_service.cc:1467] Processing CreateTablet for tablet 100545edd4ac4748b63f1f782249c369 (DEFAULT_TABLE table=test-table [id=d747f90d67ec418385149027cdbd040d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:50:58.047188 20226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 100545edd4ac4748b63f1f782249c369. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:50:58.047580 20358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 100545edd4ac4748b63f1f782249c369. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:50:58.059036 20570 tablet_bootstrap.cc:492] T 100545edd4ac4748b63f1f782249c369 P 2ab38de74582419e893b702c9dd7a051: Bootstrap starting.
I20260619 18:50:58.060985 20571 tablet_bootstrap.cc:492] T 100545edd4ac4748b63f1f782249c369 P bad6f9f3febb40e8b9161ba802d06542: Bootstrap starting.
I20260619 18:50:58.062510 20572 tablet_bootstrap.cc:492] T 100545edd4ac4748b63f1f782249c369 P 4c5611deb12845d982674dcfea33fd93: Bootstrap starting.
I20260619 18:50:58.062593 20570 tablet_bootstrap.cc:654] T 100545edd4ac4748b63f1f782249c369 P 2ab38de74582419e893b702c9dd7a051: Neither blocks nor log segments found. Creating new log.
I20260619 18:50:58.064301 20570 log.cc:824] T 100545edd4ac4748b63f1f782249c369 P 2ab38de74582419e893b702c9dd7a051: Log is configured to *not* fsync() on all Append() calls
I20260619 18:50:58.065215 20571 tablet_bootstrap.cc:654] T 100545edd4ac4748b63f1f782249c369 P bad6f9f3febb40e8b9161ba802d06542: Neither blocks nor log segments found. Creating new log.
I20260619 18:50:58.065577 20572 tablet_bootstrap.cc:654] T 100545edd4ac4748b63f1f782249c369 P 4c5611deb12845d982674dcfea33fd93: Neither blocks nor log segments found. Creating new log.
I20260619 18:50:58.066321 20571 log.cc:824] T 100545edd4ac4748b63f1f782249c369 P bad6f9f3febb40e8b9161ba802d06542: Log is configured to *not* fsync() on all Append() calls
I20260619 18:50:58.066498 20572 log.cc:824] T 100545edd4ac4748b63f1f782249c369 P 4c5611deb12845d982674dcfea33fd93: Log is configured to *not* fsync() on all Append() calls
I20260619 18:50:58.067453 20570 tablet_bootstrap.cc:492] T 100545edd4ac4748b63f1f782249c369 P 2ab38de74582419e893b702c9dd7a051: No bootstrap required, opened a new log
I20260619 18:50:58.067719 20570 ts_tablet_manager.cc:1397] T 100545edd4ac4748b63f1f782249c369 P 2ab38de74582419e893b702c9dd7a051: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260619 18:50:58.068895 20571 tablet_bootstrap.cc:492] T 100545edd4ac4748b63f1f782249c369 P bad6f9f3febb40e8b9161ba802d06542: No bootstrap required, opened a new log
I20260619 18:50:58.069115 20571 ts_tablet_manager.cc:1397] T 100545edd4ac4748b63f1f782249c369 P bad6f9f3febb40e8b9161ba802d06542: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260619 18:50:58.069355 20572 tablet_bootstrap.cc:492] T 100545edd4ac4748b63f1f782249c369 P 4c5611deb12845d982674dcfea33fd93: No bootstrap required, opened a new log
I20260619 18:50:58.069602 20572 ts_tablet_manager.cc:1397] T 100545edd4ac4748b63f1f782249c369 P 4c5611deb12845d982674dcfea33fd93: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260619 18:50:58.076057 20571 raft_consensus.cc:348] T 100545edd4ac4748b63f1f782249c369 P bad6f9f3febb40e8b9161ba802d06542 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bad6f9f3febb40e8b9161ba802d06542" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40449 } } peers { permanent_uuid: "4c5611deb12845d982674dcfea33fd93" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 38277 } } peers { permanent_uuid: "2ab38de74582419e893b702c9dd7a051" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 37273 } }
I20260619 18:50:58.076421 20571 raft_consensus.cc:374] T 100545edd4ac4748b63f1f782249c369 P bad6f9f3febb40e8b9161ba802d06542 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:50:58.076514 20571 raft_consensus.cc:729] T 100545edd4ac4748b63f1f782249c369 P bad6f9f3febb40e8b9161ba802d06542 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bad6f9f3febb40e8b9161ba802d06542, State: Initialized, Role: FOLLOWER
I20260619 18:50:58.077050 20571 consensus_queue.cc:260] T 100545edd4ac4748b63f1f782249c369 P bad6f9f3febb40e8b9161ba802d06542 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bad6f9f3febb40e8b9161ba802d06542" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40449 } } peers { permanent_uuid: "4c5611deb12845d982674dcfea33fd93" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 38277 } } peers { permanent_uuid: "2ab38de74582419e893b702c9dd7a051" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 37273 } }
I20260619 18:50:58.078007 20554 heartbeater.cc:503] Master 127.3.243.126:42879 was elected leader, sending a full tablet report...
I20260619 18:50:58.077751 20570 raft_consensus.cc:348] T 100545edd4ac4748b63f1f782249c369 P 2ab38de74582419e893b702c9dd7a051 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bad6f9f3febb40e8b9161ba802d06542" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40449 } } peers { permanent_uuid: "4c5611deb12845d982674dcfea33fd93" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 38277 } } peers { permanent_uuid: "2ab38de74582419e893b702c9dd7a051" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 37273 } }
I20260619 18:50:58.078195 20570 raft_consensus.cc:374] T 100545edd4ac4748b63f1f782249c369 P 2ab38de74582419e893b702c9dd7a051 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:50:58.078279 20570 raft_consensus.cc:729] T 100545edd4ac4748b63f1f782249c369 P 2ab38de74582419e893b702c9dd7a051 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ab38de74582419e893b702c9dd7a051, State: Initialized, Role: FOLLOWER
I20260619 18:50:58.078670 20571 ts_tablet_manager.cc:1428] T 100545edd4ac4748b63f1f782249c369 P bad6f9f3febb40e8b9161ba802d06542: Time spent starting tablet: real 0.009s user 0.008s sys 0.001s
I20260619 18:50:58.078919 20570 consensus_queue.cc:260] T 100545edd4ac4748b63f1f782249c369 P 2ab38de74582419e893b702c9dd7a051 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bad6f9f3febb40e8b9161ba802d06542" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40449 } } peers { permanent_uuid: "4c5611deb12845d982674dcfea33fd93" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 38277 } } peers { permanent_uuid: "2ab38de74582419e893b702c9dd7a051" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 37273 } }
I20260619 18:50:58.079515 20572 raft_consensus.cc:348] T 100545edd4ac4748b63f1f782249c369 P 4c5611deb12845d982674dcfea33fd93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bad6f9f3febb40e8b9161ba802d06542" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40449 } } peers { permanent_uuid: "4c5611deb12845d982674dcfea33fd93" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 38277 } } peers { permanent_uuid: "2ab38de74582419e893b702c9dd7a051" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 37273 } }
I20260619 18:50:58.080034 20423 heartbeater.cc:503] Master 127.3.243.126:42879 was elected leader, sending a full tablet report...
I20260619 18:50:58.080699 20570 ts_tablet_manager.cc:1428] T 100545edd4ac4748b63f1f782249c369 P 2ab38de74582419e893b702c9dd7a051: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260619 18:50:58.081030 20572 raft_consensus.cc:374] T 100545edd4ac4748b63f1f782249c369 P 4c5611deb12845d982674dcfea33fd93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:50:58.081133 20572 raft_consensus.cc:729] T 100545edd4ac4748b63f1f782249c369 P 4c5611deb12845d982674dcfea33fd93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c5611deb12845d982674dcfea33fd93, State: Initialized, Role: FOLLOWER
I20260619 18:50:58.081717 20572 consensus_queue.cc:260] T 100545edd4ac4748b63f1f782249c369 P 4c5611deb12845d982674dcfea33fd93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bad6f9f3febb40e8b9161ba802d06542" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40449 } } peers { permanent_uuid: "4c5611deb12845d982674dcfea33fd93" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 38277 } } peers { permanent_uuid: "2ab38de74582419e893b702c9dd7a051" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 37273 } }
I20260619 18:50:58.082751 20291 heartbeater.cc:503] Master 127.3.243.126:42879 was elected leader, sending a full tablet report...
I20260619 18:50:58.083372 20572 ts_tablet_manager.cc:1428] T 100545edd4ac4748b63f1f782249c369 P 4c5611deb12845d982674dcfea33fd93: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
W20260619 18:50:58.093298 20292 tablet.cc:2369] T 100545edd4ac4748b63f1f782249c369 P 4c5611deb12845d982674dcfea33fd93: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:50:58.127808 20576 raft_consensus.cc:482] T 100545edd4ac4748b63f1f782249c369 P bad6f9f3febb40e8b9161ba802d06542 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:50:58.128085 20576 raft_consensus.cc:504] T 100545edd4ac4748b63f1f782249c369 P bad6f9f3febb40e8b9161ba802d06542 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bad6f9f3febb40e8b9161ba802d06542" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40449 } } peers { permanent_uuid: "4c5611deb12845d982674dcfea33fd93" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 38277 } } peers { permanent_uuid: "2ab38de74582419e893b702c9dd7a051" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 37273 } }
I20260619 18:50:58.129739 20576 leader_election.cc:288] T 100545edd4ac4748b63f1f782249c369 P bad6f9f3febb40e8b9161ba802d06542 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c5611deb12845d982674dcfea33fd93 (127.3.243.65:38277), 2ab38de74582419e893b702c9dd7a051 (127.3.243.66:37273)
I20260619 18:50:58.135489 20558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:58.129988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.66:37273 (local address 127.3.243.67:35365)
0619 18:50:58.130171 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:58.130190 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 18:50:58.130311 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:50:58.130838 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:50:58.130843 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:50:58.130864 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:50:58.131046 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:50:58.131055 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:50:58.132210 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:50:58.132216 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:50:58.133652 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:50:58.133665 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:50:58.134974 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:50:58.134979 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:50:58.135110 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:58.135134 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 18:50:58.135206 (+ 72us) client_negotiation.cc:240] Negotiation successful
0619 18:50:58.135293 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260619 18:50:58.135607 20569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:58.130328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.67:35365 (local address 127.3.243.66:37273)
0619 18:50:58.130527 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0619 18:50:58.130532 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:50:58.130553 (+ 21us) server_negotiation.cc:407] Connection header received
0619 18:50:58.130625 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:50:58.130631 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:50:58.130704 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:50:58.130787 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:50:58.131185 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:58.132069 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:58.133803 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:58.134823 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:58.134985 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:58.135058 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 18:50:58.135247 (+ 189us) server_negotiation.cc:299] Negotiation successful
0619 18:50:58.135375 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 18:50:58.136315 20567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:58.130142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.67:50487 (local address 127.3.243.65:38277)
0619 18:50:58.130321 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 18:50:58.130327 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:50:58.130836 (+ 509us) server_negotiation.cc:407] Connection header received
0619 18:50:58.130973 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:50:58.130977 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:50:58.131034 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:50:58.131101 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:50:58.131572 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:58.132468 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:58.134238 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:58.135409 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:58.135552 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:58.135622 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 18:50:58.135953 (+ 331us) server_negotiation.cc:299] Negotiation successful
0619 18:50:58.136080 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 18:50:58.136382 20579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:58.129992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.65:38277 (local address 127.3.243.67:50487)
0619 18:50:58.130731 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:58.130744 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 18:50:58.130858 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:50:58.131139 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:50:58.131143 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:50:58.131166 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:50:58.131426 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:50:58.131432 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:50:58.132695 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:50:58.132699 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:50:58.134074 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:50:58.134087 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:50:58.135619 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:50:58.135625 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:50:58.135793 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:58.135821 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 18:50:58.135901 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 18:50:58.135998 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"mutex_wait_us":142,"thread_start_us":311,"threads_started":1}
I20260619 18:50:58.136576 20378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "100545edd4ac4748b63f1f782249c369" candidate_uuid: "bad6f9f3febb40e8b9161ba802d06542" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ab38de74582419e893b702c9dd7a051" is_pre_election: true
I20260619 18:50:58.137097 20378 raft_consensus.cc:2454] T 100545edd4ac4748b63f1f782249c369 P 2ab38de74582419e893b702c9dd7a051 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bad6f9f3febb40e8b9161ba802d06542 in term 0.
I20260619 18:50:58.137288 20246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "100545edd4ac4748b63f1f782249c369" candidate_uuid: "bad6f9f3febb40e8b9161ba802d06542" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c5611deb12845d982674dcfea33fd93" is_pre_election: true
I20260619 18:50:58.137717 20445 leader_election.cc:302] T 100545edd4ac4748b63f1f782249c369 P bad6f9f3febb40e8b9161ba802d06542 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ab38de74582419e893b702c9dd7a051, bad6f9f3febb40e8b9161ba802d06542; no voters:
I20260619 18:50:58.137786 20246 raft_consensus.cc:2454] T 100545edd4ac4748b63f1f782249c369 P 4c5611deb12845d982674dcfea33fd93 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bad6f9f3febb40e8b9161ba802d06542 in term 0.
I20260619 18:50:58.138041 20576 raft_consensus.cc:2781] T 100545edd4ac4748b63f1f782249c369 P bad6f9f3febb40e8b9161ba802d06542 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:50:58.138162 20576 raft_consensus.cc:482] T 100545edd4ac4748b63f1f782249c369 P bad6f9f3febb40e8b9161ba802d06542 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:50:58.138273 20576 raft_consensus.cc:3037] T 100545edd4ac4748b63f1f782249c369 P bad6f9f3febb40e8b9161ba802d06542 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:50:58.140462 20576 raft_consensus.cc:504] T 100545edd4ac4748b63f1f782249c369 P bad6f9f3febb40e8b9161ba802d06542 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bad6f9f3febb40e8b9161ba802d06542" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40449 } } peers { permanent_uuid: "4c5611deb12845d982674dcfea33fd93" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 38277 } } peers { permanent_uuid: "2ab38de74582419e893b702c9dd7a051" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 37273 } }
I20260619 18:50:58.141032 20576 leader_election.cc:288] T 100545edd4ac4748b63f1f782249c369 P bad6f9f3febb40e8b9161ba802d06542 [CANDIDATE]: Term 1 election: Requested vote from peers 4c5611deb12845d982674dcfea33fd93 (127.3.243.65:38277), 2ab38de74582419e893b702c9dd7a051 (127.3.243.66:37273)
I20260619 18:50:58.141382 20246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "100545edd4ac4748b63f1f782249c369" candidate_uuid: "bad6f9f3febb40e8b9161ba802d06542" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c5611deb12845d982674dcfea33fd93"
I20260619 18:50:58.141536 20378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "100545edd4ac4748b63f1f782249c369" candidate_uuid: "bad6f9f3febb40e8b9161ba802d06542" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ab38de74582419e893b702c9dd7a051"
I20260619 18:50:58.141577 20246 raft_consensus.cc:3037] T 100545edd4ac4748b63f1f782249c369 P 4c5611deb12845d982674dcfea33fd93 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:50:58.141728 20378 raft_consensus.cc:3037] T 100545edd4ac4748b63f1f782249c369 P 2ab38de74582419e893b702c9dd7a051 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:50:58.144618 20246 raft_consensus.cc:2454] T 100545edd4ac4748b63f1f782249c369 P 4c5611deb12845d982674dcfea33fd93 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bad6f9f3febb40e8b9161ba802d06542 in term 1.
I20260619 18:50:58.144719 20378 raft_consensus.cc:2454] T 100545edd4ac4748b63f1f782249c369 P 2ab38de74582419e893b702c9dd7a051 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bad6f9f3febb40e8b9161ba802d06542 in term 1.
I20260619 18:50:58.145047 20442 leader_election.cc:302] T 100545edd4ac4748b63f1f782249c369 P bad6f9f3febb40e8b9161ba802d06542 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c5611deb12845d982674dcfea33fd93, bad6f9f3febb40e8b9161ba802d06542; no voters:
I20260619 18:50:58.145341 20576 raft_consensus.cc:2781] T 100545edd4ac4748b63f1f782249c369 P bad6f9f3febb40e8b9161ba802d06542 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:50:58.145773 20576 raft_consensus.cc:686] T 100545edd4ac4748b63f1f782249c369 P bad6f9f3febb40e8b9161ba802d06542 [term 1 LEADER]: Becoming Leader. State: Replica: bad6f9f3febb40e8b9161ba802d06542, State: Running, Role: LEADER
I20260619 18:50:58.146227 20576 consensus_queue.cc:237] T 100545edd4ac4748b63f1f782249c369 P bad6f9f3febb40e8b9161ba802d06542 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bad6f9f3febb40e8b9161ba802d06542" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40449 } } peers { permanent_uuid: "4c5611deb12845d982674dcfea33fd93" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 38277 } } peers { permanent_uuid: "2ab38de74582419e893b702c9dd7a051" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 37273 } }
I20260619 18:50:58.150749 20106 catalog_manager.cc:5495] T 100545edd4ac4748b63f1f782249c369 P bad6f9f3febb40e8b9161ba802d06542 reported cstate change: term changed from 0 to 1, leader changed from <none> to bad6f9f3febb40e8b9161ba802d06542 (127.3.243.67). New cstate: current_term: 1 leader_uuid: "bad6f9f3febb40e8b9161ba802d06542" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bad6f9f3febb40e8b9161ba802d06542" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40449 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c5611deb12845d982674dcfea33fd93" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 38277 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ab38de74582419e893b702c9dd7a051" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 37273 } health_report { overall_health: UNKNOWN } } }
I20260619 18:50:58.166574 20566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:58.162001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55546 (local address 127.3.243.67:40449)
0619 18:50:58.162152 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 18:50:58.162156 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:50:58.162211 (+ 55us) server_negotiation.cc:407] Connection header received
0619 18:50:58.162388 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:50:58.162392 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:50:58.162473 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:50:58.162587 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:50:58.162875 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:58.163571 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:58.164894 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:58.165265 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:58.165355 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:58.165713 (+ 358us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:50:58.166190 (+ 477us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:50:58.166292 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 18:50:58.166329 (+ 37us) server_negotiation.cc:299] Negotiation successful
0619 18:50:58.166389 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 18:50:58.173938 20378 raft_consensus.cc:1261] T 100545edd4ac4748b63f1f782249c369 P 2ab38de74582419e893b702c9dd7a051 [term 1 FOLLOWER]: Refusing update from remote peer bad6f9f3febb40e8b9161ba802d06542: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:50:58.173942 20246 raft_consensus.cc:1261] T 100545edd4ac4748b63f1f782249c369 P 4c5611deb12845d982674dcfea33fd93 [term 1 FOLLOWER]: Refusing update from remote peer bad6f9f3febb40e8b9161ba802d06542: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:50:58.174849 20580 consensus_queue.cc:1035] T 100545edd4ac4748b63f1f782249c369 P bad6f9f3febb40e8b9161ba802d06542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c5611deb12845d982674dcfea33fd93" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 38277 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:50:58.175151 20576 consensus_queue.cc:1035] T 100545edd4ac4748b63f1f782249c369 P bad6f9f3febb40e8b9161ba802d06542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ab38de74582419e893b702c9dd7a051" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 37273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:50:58.190788 20584 mvcc.cc:205] Tried to move back new op lower bound from 7298642158271524864 to 7298642158170566656. Current Snapshot: MvccSnapshot[applied={T|T < 7298642158271524864 or (T in {7298642158271524864})}]
I20260619 18:50:58.196859 20106 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45222:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:50:58.197113 20106 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45222:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:50:58.200314 20106 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5eb6468c42124bb8acf48ba3fbceac41: Sending DeleteTablet for 3 replicas of tablet 100545edd4ac4748b63f1f782249c369
I20260619 18:50:58.201295 20358 tablet_service.cc:1514] Processing DeleteTablet for tablet 100545edd4ac4748b63f1f782249c369 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:50:58 UTC) from {username='kudu', principal='kudu/127.3.243.126@KRBTEST.COM'} at 127.0.0.1:60004
I20260619 18:50:58.201327 20226 tablet_service.cc:1514] Processing DeleteTablet for tablet 100545edd4ac4748b63f1f782249c369 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:50:58 UTC) from {username='kudu', principal='kudu/127.3.243.126@KRBTEST.COM'} at 127.0.0.1:49464
I20260619 18:50:58.201566 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 20143
I20260619 18:50:58.202153 20489 tablet_service.cc:1514] Processing DeleteTablet for tablet 100545edd4ac4748b63f1f782249c369 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:50:58 UTC) from {username='kudu', principal='kudu/127.3.243.126@KRBTEST.COM'} at 127.0.0.1:55544
I20260619 18:50:58.203377 20592 tablet_replica.cc:330] T 100545edd4ac4748b63f1f782249c369 P 4c5611deb12845d982674dcfea33fd93: stopping tablet replica
I20260619 18:50:58.203611 20593 tablet_replica.cc:330] T 100545edd4ac4748b63f1f782249c369 P 2ab38de74582419e893b702c9dd7a051: stopping tablet replica
I20260619 18:50:58.204026 20592 raft_consensus.cc:2229] T 100545edd4ac4748b63f1f782249c369 P 4c5611deb12845d982674dcfea33fd93 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:50:58.205408 20593 raft_consensus.cc:2229] T 100545edd4ac4748b63f1f782249c369 P 2ab38de74582419e893b702c9dd7a051 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:50:58.205884 20593 raft_consensus.cc:2258] T 100545edd4ac4748b63f1f782249c369 P 2ab38de74582419e893b702c9dd7a051 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:50:58.206365 20592 raft_consensus.cc:2258] T 100545edd4ac4748b63f1f782249c369 P 4c5611deb12845d982674dcfea33fd93 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:50:58.209173 20592 ts_tablet_manager.cc:1905] T 100545edd4ac4748b63f1f782249c369 P 4c5611deb12845d982674dcfea33fd93: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:50:58.210402 20593 ts_tablet_manager.cc:1905] T 100545edd4ac4748b63f1f782249c369 P 2ab38de74582419e893b702c9dd7a051: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:50:58.213027 20594 tablet_replica.cc:330] T 100545edd4ac4748b63f1f782249c369 P bad6f9f3febb40e8b9161ba802d06542: stopping tablet replica
I20260619 18:50:58.213521 20594 raft_consensus.cc:2229] T 100545edd4ac4748b63f1f782249c369 P bad6f9f3febb40e8b9161ba802d06542 [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:50:58.214182 20594 raft_consensus.cc:2258] T 100545edd4ac4748b63f1f782249c369 P bad6f9f3febb40e8b9161ba802d06542 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:50:58.215942 20594 ts_tablet_manager.cc:1905] T 100545edd4ac4748b63f1f782249c369 P bad6f9f3febb40e8b9161ba802d06542: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:50:58.216890 20567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:58.202465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49476 (local address 127.3.243.65:38277)
0619 18:50:58.202613 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 18:50:58.202618 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:50:58.202646 (+ 28us) server_negotiation.cc:407] Connection header received
0619 18:50:58.202695 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:50:58.202699 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:50:58.202765 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:50:58.202841 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:50:58.204508 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:58.205207 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:58.207027 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:58.207412 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:58.207516 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:58.211027 (+ 3511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:50:58.211039 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:50:58.211061 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:50:58.211094 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:50:58.214117 (+ 3023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:58.214872 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:58.214880 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:58.214883 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:58.215030 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:58.215395 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:58.215400 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:58.215403 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:58.215815 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:50:58.216095 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0619 18:50:58.216380 (+ 285us) server_negotiation.cc:299] Negotiation successful
0619 18:50:58.216569 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 18:50:58.217680 20286 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:50:58.220391 20593 ts_tablet_manager.cc:1918] T 100545edd4ac4748b63f1f782249c369 P 2ab38de74582419e893b702c9dd7a051: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:50:58.220579 20593 log.cc:1196] T 100545edd4ac4748b63f1f782249c369 P 2ab38de74582419e893b702c9dd7a051: Deleting WAL directory at /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-1/wal/wals/100545edd4ac4748b63f1f782249c369
I20260619 18:50:58.221042 20593 ts_tablet_manager.cc:1939] T 100545edd4ac4748b63f1f782249c369 P 2ab38de74582419e893b702c9dd7a051: Deleting consensus metadata
I20260619 18:50:58.222510 20093 catalog_manager.cc:4841] TS 2ab38de74582419e893b702c9dd7a051 (127.3.243.66:37273): tablet 100545edd4ac4748b63f1f782249c369 (table test-table [id=d747f90d67ec418385149027cdbd040d]) successfully deleted
I20260619 18:50:58.226892 20594 ts_tablet_manager.cc:1918] T 100545edd4ac4748b63f1f782249c369 P bad6f9f3febb40e8b9161ba802d06542: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:50:58.227052 20594 log.cc:1196] T 100545edd4ac4748b63f1f782249c369 P bad6f9f3febb40e8b9161ba802d06542: Deleting WAL directory at /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-2/wal/wals/100545edd4ac4748b63f1f782249c369
I20260619 18:50:58.227617 20594 ts_tablet_manager.cc:1939] T 100545edd4ac4748b63f1f782249c369 P bad6f9f3febb40e8b9161ba802d06542: Deleting consensus metadata
I20260619 18:50:58.229336 20092 catalog_manager.cc:4841] TS bad6f9f3febb40e8b9161ba802d06542 (127.3.243.67:40449): tablet 100545edd4ac4748b63f1f782249c369 (table test-table [id=d747f90d67ec418385149027cdbd040d]) successfully deleted
I20260619 18:50:58.308917 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 20143
W20260619 18:50:58.322979 20090 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.243.65:38277 (error 108)
I20260619 18:50:58.323163 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 20295
I20260619 18:50:58.325028 20564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:58.324552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.65:38277 (local address 127.0.0.1:49486)
0619 18:50:58.324738 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:58.324858 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.65:38277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260619 18:50:58.325273 20090 catalog_manager.cc:4584] TS 4c5611deb12845d982674dcfea33fd93 (127.3.243.65:38277): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 100545edd4ac4748b63f1f782249c369: Network error: Client connection negotiation failed: client connection to 127.3.243.65:38277: connect: Connection refused (error 111)
I20260619 18:50:58.339485 20569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:58.323961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60012 (local address 127.3.243.66:37273)
0619 18:50:58.324134 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 18:50:58.324140 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:50:58.324215 (+ 75us) server_negotiation.cc:407] Connection header received
0619 18:50:58.324432 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:50:58.324437 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:50:58.324526 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:50:58.324631 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:50:58.326915 (+ 2284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:58.327649 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:58.329445 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:58.329848 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:58.329937 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:58.333294 (+ 3357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:50:58.333307 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:50:58.333322 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:50:58.333366 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:50:58.336768 (+ 3402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:58.337620 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:58.337627 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:58.337629 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:58.337768 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:58.338174 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:58.338178 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:58.338180 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:58.338531 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:50:58.338757 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0619 18:50:58.339064 (+ 307us) server_negotiation.cc:299] Negotiation successful
0619 18:50:58.339219 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 18:50:58.340360 20418 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:50:58.378975 20564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:58.378428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.65:38277 (local address 127.0.0.1:49488)
0619 18:50:58.378657 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:58.378778 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.65:38277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260619 18:50:58.380128 20564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:58.379729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.65:38277 (local address 127.0.0.1:49490)
0619 18:50:58.379889 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:58.379977 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.65:38277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260619 18:50:58.428447 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 20295
I20260619 18:50:58.440793 20564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:58.440340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.65:38277 (local address 127.0.0.1:49504)
0619 18:50:58.440541 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:58.440629 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.65:38277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260619 18:50:58.441831 20564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:58.441502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.65:38277 (local address 127.0.0.1:49506)
0619 18:50:58.441648 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:58.441717 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.65:38277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260619 18:50:58.448529 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 20426
I20260619 18:50:58.462683 20566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:58.449276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55552 (local address 127.3.243.67:40449)
0619 18:50:58.449468 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 18:50:58.449473 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:50:58.449498 (+ 25us) server_negotiation.cc:407] Connection header received
0619 18:50:58.449597 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:50:58.449603 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:50:58.449704 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:50:58.449792 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:50:58.451096 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:58.451879 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:58.453346 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:50:58.453807 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:50:58.453924 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:50:58.456551 (+ 2627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:50:58.456562 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:50:58.456576 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:50:58.456615 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:50:58.459739 (+ 3124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:58.460513 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:58.460522 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:58.460524 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:58.460683 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:50:58.461077 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:50:58.461080 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:50:58.461082 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:50:58.461463 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:50:58.461775 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0619 18:50:58.462090 (+ 315us) server_negotiation.cc:299] Negotiation successful
0619 18:50:58.462317 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 18:50:58.463539 20549 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:50:58.522485 20564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:58.521918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.65:38277 (local address 127.0.0.1:49508)
0619 18:50:58.522182 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:58.522301 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.65:38277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260619 18:50:58.523514 20564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:58.523109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.65:38277 (local address 127.0.0.1:49512)
0619 18:50:58.523298 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:58.523384 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.65:38277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260619 18:50:58.582504 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 20426
I20260619 18:50:58.599141 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 20073
I20260619 18:50:58.600251 20136 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:50:58.716782 20564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:50:58.716112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.65:38277 (local address 127.0.0.1:49518)
0619 18:50:58.716467 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0619 18:50:58.716586 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.65:38277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211}
I20260619 18:50:58.716817 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 20073
I20260619 18:51:18.734303 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.126:42879
--webserver_interface=127.3.243.126
--webserver_port=46413
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab
--principal=kudu/127.3.243.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:45379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.126:42879
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:51:19.054284 20600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:51:19.054595 20600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:51:19.054728 20600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:51:19.061673 20600 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 18:51:19.061765 20600 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 18:51:19.061827 20600 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 18:51:19.061899 20600 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:51:19.061971 20600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:51:19.062062 20600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:51:19.062121 20600 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:51:19.062193 20600 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:51:19.071615 20600 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:45379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.3.243.126:42879
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab
--principal=kudu/127.3.243.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.126:42879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.243.126
--webserver_port=46413
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:51:19.072839 20600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:51:19.074090 20600 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:51:19.085371 20605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:51:19.085345 20608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:51:19.085386 20606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:51:19.086009 20600 server_base.cc:1034] running on GCE node
I20260619 18:51:19.086687 20600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:51:19.088075 20600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:51:19.089300 20600 hybrid_clock.cc:648] HybridClock initialized: now 1781895079089235 us; error 37 us; skew 500 ppm
I20260619 18:51:19.089638 20600 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:51:19 dist-test-slave-m9wg krb5kdc[19523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895079, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:51:19.097514 20600 init.cc:377] Logged in from keytab as kudu/127.3.243.126@KRBTEST.COM (short username kudu)
I20260619 18:51:19.098996 20600 webserver.cc:460] Webserver started at http://127.3.243.126:46413/ using document root <none> and password file <none>
I20260619 18:51:19.099668 20600 fs_manager.cc:359] Metadata directory not provided
I20260619 18:51:19.099766 20600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:51:19.105286 20600 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.002s
I20260619 18:51:19.108125 20615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:51:19.109406 20600 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 18:51:19.109567 20600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/master-0/wal
uuid: "5eb6468c42124bb8acf48ba3fbceac41"
format_stamp: "Formatted at 2026-06-19 18:50:33 on dist-test-slave-m9wg"
server_key: "5bee5c99c932f10c288f5b8278cc5e63"
server_key_iv: "9b5ff271c13b1f6b55a7ae9a06a2e25f"
server_key_version: "encryptionkey@0"
I20260619 18:51:19.110509 20600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:51:19.124758 20600 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:51:19.128618 20600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:51:19.128872 20600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:51:19.145885 20600 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.126:42879
I20260619 18:51:19.145908 20666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.126:42879 every 8 connection(s)
I20260619 18:51:19.147343 20600 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
I20260619 18:51:19.154058 20667 sys_catalog.cc:263] Verifying existing consensus state
I20260619 18:51:19.155435 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 20600
I20260619 18:51:19.156484 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.65:38277
--local_ip_for_outbound_sockets=127.3.243.65
--tserver_master_addrs=127.3.243.126:42879
--webserver_port=34305
--webserver_interface=127.3.243.65
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.65.keytab
--principal=kudu/127.3.243.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:45379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 18:51:19.158257 20667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5eb6468c42124bb8acf48ba3fbceac41: Bootstrap starting.
I20260619 18:51:19.179908 20667 log.cc:824] T 00000000000000000000000000000000 P 5eb6468c42124bb8acf48ba3fbceac41: Log is configured to *not* fsync() on all Append() calls
I20260619 18:51:19.195626 20667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5eb6468c42124bb8acf48ba3fbceac41: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 18:51:19.196127 20667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5eb6468c42124bb8acf48ba3fbceac41: Bootstrap complete.
I20260619 18:51:19.204995 20667 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5eb6468c42124bb8acf48ba3fbceac41 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5eb6468c42124bb8acf48ba3fbceac41" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 42879 } }
I20260619 18:51:19.205879 20667 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5eb6468c42124bb8acf48ba3fbceac41 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5eb6468c42124bb8acf48ba3fbceac41, State: Initialized, Role: FOLLOWER
I20260619 18:51:19.206434 20667 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5eb6468c42124bb8acf48ba3fbceac41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5eb6468c42124bb8acf48ba3fbceac41" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 42879 } }
I20260619 18:51:19.206642 20667 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5eb6468c42124bb8acf48ba3fbceac41 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:51:19.206755 20667 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5eb6468c42124bb8acf48ba3fbceac41 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:51:19.206900 20667 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5eb6468c42124bb8acf48ba3fbceac41 [term 2 FOLLOWER]: Advancing to term 3
I20260619 18:51:19.210536 20667 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5eb6468c42124bb8acf48ba3fbceac41 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5eb6468c42124bb8acf48ba3fbceac41" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 42879 } }
I20260619 18:51:19.210978 20667 leader_election.cc:302] T 00000000000000000000000000000000 P 5eb6468c42124bb8acf48ba3fbceac41 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5eb6468c42124bb8acf48ba3fbceac41; no voters:
I20260619 18:51:19.211598 20667 leader_election.cc:288] T 00000000000000000000000000000000 P 5eb6468c42124bb8acf48ba3fbceac41 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 18:51:19.211730 20671 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5eb6468c42124bb8acf48ba3fbceac41 [term 3 FOLLOWER]: Leader election won for term 3
I20260619 18:51:19.212788 20671 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5eb6468c42124bb8acf48ba3fbceac41 [term 3 LEADER]: Becoming Leader. State: Replica: 5eb6468c42124bb8acf48ba3fbceac41, State: Running, Role: LEADER
I20260619 18:51:19.213460 20671 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5eb6468c42124bb8acf48ba3fbceac41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5eb6468c42124bb8acf48ba3fbceac41" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 42879 } }
I20260619 18:51:19.213744 20667 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5eb6468c42124bb8acf48ba3fbceac41 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:51:19.217612 20672 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5eb6468c42124bb8acf48ba3fbceac41 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "5eb6468c42124bb8acf48ba3fbceac41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5eb6468c42124bb8acf48ba3fbceac41" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 42879 } } }
I20260619 18:51:19.217634 20673 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5eb6468c42124bb8acf48ba3fbceac41 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5eb6468c42124bb8acf48ba3fbceac41. Latest consensus state: current_term: 3 leader_uuid: "5eb6468c42124bb8acf48ba3fbceac41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5eb6468c42124bb8acf48ba3fbceac41" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 42879 } } }
I20260619 18:51:19.217998 20672 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5eb6468c42124bb8acf48ba3fbceac41 [sys.catalog]: This master's current role is: LEADER
I20260619 18:51:19.217995 20673 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5eb6468c42124bb8acf48ba3fbceac41 [sys.catalog]: This master's current role is: LEADER
I20260619 18:51:19.219048 20679 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:51:19.235674 20679 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:51:19.237200 20679 catalog_manager.cc:1259] Loaded cluster ID: c15cbeedde704a27b2d2b3bbd371d776
I20260619 18:51:19.237382 20679 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:51:19.241343 20679 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:51:19.243083 20679 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5eb6468c42124bb8acf48ba3fbceac41: Loaded TSK: 0
I20260619 18:51:19.243337 20679 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5eb6468c42124bb8acf48ba3fbceac41: Loaded TSK: 1
F20260619 18:51:19.244524 20679 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fabb9fe69ad google::LogMessage::Fail() at ??:0
@ 0x7fabb9fe988a google::LogMessage::SendToLog() at ??:0
@ 0x7fabb9fe64de google::LogMessage::Flush() at ??:0
@ 0x7fabb9fe7a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fabbdd4a7d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fabbdd3973e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fabcfa151c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fabcfa14783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fabcfaa02a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7fabcfaa00c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7fabcfaea5e9 std::function<>::operator()() at ??:0
@ 0x7fabcfa18cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fabcfa16c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fabcfa9e610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fabcfa9e439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fabbcc32345 kudu::ThreadPool::DispatchThread() at ??:0
I20260619 18:51:19.302495 20691 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7fabbcc3591c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fabbcc35799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fabbcc10eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fabbfeea6db start_thread at ??:0
@ 0x7fabb8e8061f clone at ??:0
W20260619 18:51:19.517921 20669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:51:19.518213 20669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:51:19.518352 20669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:51:19.525921 20669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:51:19.526041 20669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:51:19.526187 20669 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.65
I20260619 18:51:19.535827 20669 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:45379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.65.keytab
--principal=kudu/127.3.243.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.65:38277
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.243.65
--webserver_port=34305
--tserver_master_addrs=127.3.243.126:42879
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.65
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:51:19.537108 20669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:51:19.538432 20669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:51:19.551116 20715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:51:19.551081 20717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:51:19.551578 20669 server_base.cc:1034] running on GCE node
W20260619 18:51:19.551157 20714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:51:19.552277 20669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:51:19.553133 20669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:51:19.554338 20669 hybrid_clock.cc:648] HybridClock initialized: now 1781895079554280 us; error 44 us; skew 500 ppm
I20260619 18:51:19.554639 20669 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:51:19 dist-test-slave-m9wg krb5kdc[19523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895079, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:51:19.557583 20669 init.cc:377] Logged in from keytab as kudu/127.3.243.65@KRBTEST.COM (short username kudu)
I20260619 18:51:19.558947 20669 webserver.cc:460] Webserver started at http://127.3.243.65:34305/ using document root <none> and password file <none>
I20260619 18:51:19.559588 20669 fs_manager.cc:359] Metadata directory not provided
I20260619 18:51:19.559703 20669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:51:19.565083 20669 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 18:51:19.567876 20724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:51:19.569190 20669 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 18:51:19.569348 20669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-0/wal
uuid: "4c5611deb12845d982674dcfea33fd93"
format_stamp: "Formatted at 2026-06-19 18:50:34 on dist-test-slave-m9wg"
server_key: "8949a1fc82148059d2455b1dfd58e8bb"
server_key_iv: "eb1da66fec32d889e1dc8f419c86652b"
server_key_version: "encryptionkey@0"
I20260619 18:51:19.570294 20669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:51:19.593041 20669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:51:19.596922 20669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:51:19.597177 20669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:51:19.598206 20669 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:51:19.599934 20731 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260619 18:51:19.605489 20669 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 18:51:19.605612 20669 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260619 18:51:19.605726 20669 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 18:51:19.608793 20669 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 18:51:19.608879 20669 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.003s sys 0.000s
I20260619 18:51:19.609207 20731 tablet_bootstrap.cc:492] T 100545edd4ac4748b63f1f782249c369 P 4c5611deb12845d982674dcfea33fd93: Bootstrap starting.
I20260619 18:51:19.617202 20731 log.cc:824] T 100545edd4ac4748b63f1f782249c369 P 4c5611deb12845d982674dcfea33fd93: Log is configured to *not* fsync() on all Append() calls
I20260619 18:51:19.631623 20731 tablet_bootstrap.cc:492] T 100545edd4ac4748b63f1f782249c369 P 4c5611deb12845d982674dcfea33fd93: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 18:51:19.632256 20731 tablet_bootstrap.cc:492] T 100545edd4ac4748b63f1f782249c369 P 4c5611deb12845d982674dcfea33fd93: Bootstrap complete.
I20260619 18:51:19.632861 20731 ts_tablet_manager.cc:1397] T 100545edd4ac4748b63f1f782249c369 P 4c5611deb12845d982674dcfea33fd93: Time spent bootstrapping tablet: real 0.024s user 0.018s sys 0.004s
I20260619 18:51:19.642838 20731 raft_consensus.cc:348] T 100545edd4ac4748b63f1f782249c369 P 4c5611deb12845d982674dcfea33fd93 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bad6f9f3febb40e8b9161ba802d06542" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40449 } } peers { permanent_uuid: "4c5611deb12845d982674dcfea33fd93" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 38277 } } peers { permanent_uuid: "2ab38de74582419e893b702c9dd7a051" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 37273 } }
I20260619 18:51:19.644942 20731 raft_consensus.cc:729] T 100545edd4ac4748b63f1f782249c369 P 4c5611deb12845d982674dcfea33fd93 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c5611deb12845d982674dcfea33fd93, State: Initialized, Role: FOLLOWER
I20260619 18:51:19.645761 20731 consensus_queue.cc:260] T 100545edd4ac4748b63f1f782249c369 P 4c5611deb12845d982674dcfea33fd93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bad6f9f3febb40e8b9161ba802d06542" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40449 } } peers { permanent_uuid: "4c5611deb12845d982674dcfea33fd93" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 38277 } } peers { permanent_uuid: "2ab38de74582419e893b702c9dd7a051" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 37273 } }
I20260619 18:51:19.647668 20731 ts_tablet_manager.cc:1428] T 100545edd4ac4748b63f1f782249c369 P 4c5611deb12845d982674dcfea33fd93: Time spent starting tablet: real 0.015s user 0.009s sys 0.004s
I20260619 18:51:19.658361 20669 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.65:38277
I20260619 18:51:19.658397 20839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.65:38277 every 8 connection(s)
I20260619 18:51:19.659970 20669 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-0/data/info.pb
I20260619 18:51:19.663746 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 20669
I20260619 18:51:19.664590 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.66:37273
--local_ip_for_outbound_sockets=127.3.243.66
--tserver_master_addrs=127.3.243.126:42879
--webserver_port=36089
--webserver_interface=127.3.243.66
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.66.keytab
--principal=kudu/127.3.243.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:45379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 18:51:19.665159 20842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:19.662726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:42879 (local address 127.3.243.65:59879)
0619 18:51:19.663596 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:19.663877 (+ 281us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:42879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":298,"threads_started":1}
W20260619 18:51:19.665874 20840 heartbeater.cc:644] Failed to heartbeat to 127.3.243.126:42879 (0 consecutive failures): Network error: Failed to ping master at 127.3.243.126:42879: Client connection negotiation failed: client connection to 127.3.243.126:42879: connect: Connection refused (error 111)
I20260619 18:51:19.666733 20842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:19.666379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:42879 (local address 127.3.243.65:44963)
0619 18:51:19.666535 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:19.666615 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:42879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260619 18:51:19.667622 20842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:19.667274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:42879 (local address 127.3.243.65:60761)
0619 18:51:19.667412 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:19.667494 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:42879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260619 18:51:19.667837 20840 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 18:51:19.977285 20843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:51:19.977579 20843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:51:19.977718 20843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:51:19.984983 20843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:51:19.985117 20843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:51:19.985273 20843 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.66
I20260619 18:51:19.994933 20843 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:45379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.66.keytab
--principal=kudu/127.3.243.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.66:37273
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.243.66
--webserver_port=36089
--tserver_master_addrs=127.3.243.126:42879
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.66
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:51:19.996176 20843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:51:19.997447 20843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:51:20.009704 20848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:51:20.009727 20851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:51:20.009761 20849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:51:20.010715 20843 server_base.cc:1034] running on GCE node
I20260619 18:51:20.011245 20843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:51:20.012116 20843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:51:20.013340 20843 hybrid_clock.cc:648] HybridClock initialized: now 1781895080013298 us; error 35 us; skew 500 ppm
I20260619 18:51:20.013625 20843 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:51:20 dist-test-slave-m9wg krb5kdc[19523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895080, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:51:20.016525 20843 init.cc:377] Logged in from keytab as kudu/127.3.243.66@KRBTEST.COM (short username kudu)
I20260619 18:51:20.017870 20843 webserver.cc:460] Webserver started at http://127.3.243.66:36089/ using document root <none> and password file <none>
I20260619 18:51:20.018523 20843 fs_manager.cc:359] Metadata directory not provided
I20260619 18:51:20.018631 20843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:51:20.023686 20843 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 18:51:20.026257 20858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:51:20.027565 20843 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 18:51:20.027715 20843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-1/wal
uuid: "2ab38de74582419e893b702c9dd7a051"
format_stamp: "Formatted at 2026-06-19 18:50:34 on dist-test-slave-m9wg"
server_key: "2e62b509e7add8e89f6a640ab71367ab"
server_key_iv: "4bdd0a384976c71967d5e44491c4d209"
server_key_version: "encryptionkey@0"
I20260619 18:51:20.028731 20843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:51:20.041872 20843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:51:20.045552 20843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:51:20.045783 20843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:51:20.046708 20843 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:51:20.048009 20843 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:51:20.048103 20843 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:51:20.048218 20843 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:51:20.048288 20843 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:51:20.078228 20843 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.66:37273
I20260619 18:51:20.078298 20970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.66:37273 every 8 connection(s)
I20260619 18:51:20.079607 20843 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-1/data/info.pb
I20260619 18:51:20.084187 20973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:20.082416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:42879 (local address 127.3.243.66:40285)
0619 18:51:20.083098 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:20.083323 (+ 225us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:42879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":261,"threads_started":1}
W20260619 18:51:20.084761 20971 heartbeater.cc:644] Failed to heartbeat to 127.3.243.126:42879 (0 consecutive failures): Network error: Failed to ping master at 127.3.243.126:42879: Client connection negotiation failed: client connection to 127.3.243.126:42879: connect: Connection refused (error 111)
I20260619 18:51:20.085458 20973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:20.085171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:42879 (local address 127.3.243.66:40921)
0619 18:51:20.085292 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:20.085354 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:42879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260619 18:51:20.086192 20973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:20.085897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:42879 (local address 127.3.243.66:59195)
0619 18:51:20.086028 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:20.086096 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:42879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260619 18:51:20.086419 20971 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 18:51:20.086642 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 20843
I20260619 18:51:20.087229 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.67:40449
--local_ip_for_outbound_sockets=127.3.243.67
--tserver_master_addrs=127.3.243.126:42879
--webserver_port=45589
--webserver_interface=127.3.243.67
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.67.keytab
--principal=kudu/127.3.243.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:45379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:51:20.401149 20974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:51:20.401410 20974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:51:20.401511 20974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:51:20.408576 20974 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:51:20.408669 20974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:51:20.408784 20974 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.67
I20260619 18:51:20.417968 20974 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:45379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.67.keytab
--principal=kudu/127.3.243.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.67:40449
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.243.67
--webserver_port=45589
--tserver_master_addrs=127.3.243.126:42879
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.67
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:51:20.419121 20974 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:51:20.420334 20974 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:51:20.432495 20979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:51:20.432471 20980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:51:20.432684 20982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:51:20.433435 20974 server_base.cc:1034] running on GCE node
I20260619 18:51:20.433970 20974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:51:20.434841 20974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:51:20.436034 20974 hybrid_clock.cc:648] HybridClock initialized: now 1781895080435998 us; error 27 us; skew 500 ppm
I20260619 18:51:20.436342 20974 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:51:20 dist-test-slave-m9wg krb5kdc[19523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895080, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:51:20.439167 20974 init.cc:377] Logged in from keytab as kudu/127.3.243.67@KRBTEST.COM (short username kudu)
I20260619 18:51:20.440562 20974 webserver.cc:460] Webserver started at http://127.3.243.67:45589/ using document root <none> and password file <none>
I20260619 18:51:20.441174 20974 fs_manager.cc:359] Metadata directory not provided
I20260619 18:51:20.441289 20974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:51:20.446295 20974 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260619 18:51:20.448985 20989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:51:20.450227 20974 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:51:20.450367 20974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-2/wal
uuid: "bad6f9f3febb40e8b9161ba802d06542"
format_stamp: "Formatted at 2026-06-19 18:50:35 on dist-test-slave-m9wg"
server_key: "3cedacdc2802ab3a8ff6cf01cef22bf4"
server_key_iv: "d46435461ac1345cae481d52c8f3b7e8"
server_key_version: "encryptionkey@0"
I20260619 18:51:20.451272 20974 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:51:20.477557 20974 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:51:20.481428 20974 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:51:20.481683 20974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:51:20.482617 20974 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:51:20.483875 20974 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:51:20.483974 20974 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:51:20.484081 20974 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:51:20.484169 20974 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:51:20.513859 20974 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.67:40449
I20260619 18:51:20.513914 21101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.67:40449 every 8 connection(s)
I20260619 18:51:20.515251 20974 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894810715282-4045-0/minicluster-data/ts-2/data/info.pb
I20260619 18:51:20.518734 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 20974
I20260619 18:51:20.519778 21104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:20.518015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:42879 (local address 127.3.243.67:60871)
0619 18:51:20.518697 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:20.518958 (+ 261us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:42879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":261,"threads_started":1}
W20260619 18:51:20.520394 21102 heartbeater.cc:644] Failed to heartbeat to 127.3.243.126:42879 (0 consecutive failures): Network error: Failed to ping master at 127.3.243.126:42879: Client connection negotiation failed: client connection to 127.3.243.126:42879: connect: Connection refused (error 111)
I20260619 18:51:20.521131 21104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:20.520833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:42879 (local address 127.3.243.67:56837)
0619 18:51:20.520965 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:20.521028 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:42879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260619 18:51:20.521848 21104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:20.521561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:42879 (local address 127.3.243.67:60901)
0619 18:51:20.521684 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:20.521750 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:42879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260619 18:51:20.522047 21102 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 18:51:20.669077 21106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:20.668353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:42879 (local address 127.3.243.65:37943)
0619 18:51:20.668785 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:20.668871 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:42879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":210,"threads_started":1}
I20260619 18:51:21.087690 21107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:21.086947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:42879 (local address 127.3.243.66:36937)
0619 18:51:21.087372 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:21.087474 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:42879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":199,"threads_started":1}
I20260619 18:51:21.216569 21108 raft_consensus.cc:482] T 100545edd4ac4748b63f1f782249c369 P 4c5611deb12845d982674dcfea33fd93 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:51:21.216840 21108 raft_consensus.cc:504] T 100545edd4ac4748b63f1f782249c369 P 4c5611deb12845d982674dcfea33fd93 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bad6f9f3febb40e8b9161ba802d06542" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40449 } } peers { permanent_uuid: "4c5611deb12845d982674dcfea33fd93" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 38277 } } peers { permanent_uuid: "2ab38de74582419e893b702c9dd7a051" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 37273 } }
I20260619 18:51:21.218377 21108 leader_election.cc:288] T 100545edd4ac4748b63f1f782249c369 P 4c5611deb12845d982674dcfea33fd93 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bad6f9f3febb40e8b9161ba802d06542 (127.3.243.67:40449), 2ab38de74582419e893b702c9dd7a051 (127.3.243.66:37273)
Jun 19 18:51:21 dist-test-slave-m9wg krb5kdc[19523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895079, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.65@KRBTEST.COM for kudu/127.3.243.66@KRBTEST.COM
Jun 19 18:51:21 dist-test-slave-m9wg krb5kdc[19523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895079, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.65@KRBTEST.COM for kudu/127.3.243.67@KRBTEST.COM
I20260619 18:51:21.250130 21111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:21.218818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.65:60815 (local address 127.3.243.67:40449)
0619 18:51:21.229469 (+ 10651us) server_negotiation.cc:206] Beginning negotiation
0619 18:51:21.229486 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 18:51:21.230003 (+ 517us) server_negotiation.cc:407] Connection header received
0619 18:51:21.231917 (+ 1914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:51:21.231926 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:51:21.232332 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:51:21.232605 (+ 273us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:51:21.234470 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:21.235676 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:21.238233 (+ 2557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:21.239191 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:21.239361 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:21.243656 (+ 4295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:51:21.243663 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:51:21.243674 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:51:21.243712 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:51:21.246536 (+ 2824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:51:21.247349 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:51:21.247355 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:51:21.247361 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:51:21.247521 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:51:21.247956 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:51:21.247960 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:51:21.247962 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:51:21.248477 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:51:21.248652 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0619 18:51:21.249405 (+ 753us) server_negotiation.cc:299] Negotiation successful
0619 18:51:21.249727 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10466,"thread_start_us":10337,"threads_started":1}
I20260619 18:51:21.250155 21112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:21.219125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.65:46317 (local address 127.3.243.66:37273)
0619 18:51:21.229187 (+ 10062us) server_negotiation.cc:206] Beginning negotiation
0619 18:51:21.229205 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 18:51:21.229923 (+ 718us) server_negotiation.cc:407] Connection header received
0619 18:51:21.231937 (+ 2014us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:51:21.231945 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:51:21.232315 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:51:21.232633 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:51:21.234473 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:21.235679 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:21.238233 (+ 2554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:21.239191 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:21.239360 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:21.243503 (+ 4143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:51:21.243511 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:51:21.243524 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:51:21.243559 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:51:21.246377 (+ 2818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:51:21.247375 (+ 998us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:51:21.247382 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:51:21.247388 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:51:21.247544 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:51:21.247980 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:51:21.247986 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:51:21.247989 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:51:21.248516 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:51:21.248685 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0619 18:51:21.249430 (+ 745us) server_negotiation.cc:299] Negotiation successful
0619 18:51:21.249799 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9840,"thread_start_us":9670,"threads_started":1}
I20260619 18:51:21.250149 21110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:21.218597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.66:37273 (local address 127.3.243.65:46317)
0619 18:51:21.229750 (+ 11153us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:21.229782 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0619 18:51:21.230304 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:51:21.232710 (+ 2406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:51:21.232717 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:51:21.233580 (+ 863us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:51:21.234286 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:51:21.234320 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:51:21.235863 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:51:21.235867 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:51:21.238080 (+ 2213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:51:21.238094 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:51:21.239414 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:51:21.239419 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:51:21.239727 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:21.240520 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:51:21.240550 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:51:21.243239 (+ 2689us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:51:21.246547 (+ 3308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:51:21.246556 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:51:21.246569 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:51:21.247228 (+ 659us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:51:21.247657 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:51:21.247661 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:51:21.247664 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:51:21.247863 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:51:21.248636 (+ 773us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:51:21.248641 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:51:21.248933 (+ 292us) client_negotiation.cc:769] Sending connection context
0619 18:51:21.249155 (+ 222us) client_negotiation.cc:240] Negotiation successful
0619 18:51:21.249676 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11012,"mutex_wait_us":269,"spinlock_wait_cycles":37504,"thread_start_us":11088,"threads_started":1}
I20260619 18:51:21.250526 21109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:21.218619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.67:40449 (local address 127.3.243.65:60815)
0619 18:51:21.229915 (+ 11296us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:21.229928 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 18:51:21.230292 (+ 364us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:51:21.232707 (+ 2415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:51:21.232717 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:51:21.233601 (+ 884us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:51:21.234288 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:51:21.234319 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:51:21.235858 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:51:21.235864 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:51:21.238082 (+ 2218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:51:21.238094 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:51:21.239395 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:51:21.239402 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:51:21.239721 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:21.240559 (+ 838us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:51:21.240577 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:51:21.243428 (+ 2851us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:51:21.246663 (+ 3235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:51:21.246669 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:51:21.246673 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:51:21.247221 (+ 548us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:51:21.247629 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:51:21.247634 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:51:21.247638 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:51:21.247847 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:51:21.248614 (+ 767us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:51:21.248622 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:51:21.248934 (+ 312us) client_negotiation.cc:769] Sending connection context
0619 18:51:21.249155 (+ 221us) client_negotiation.cc:240] Negotiation successful
0619 18:51:21.249836 (+ 681us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11202,"mutex_wait_us":527,"spinlock_wait_cycles":38272,"thread_start_us":10847,"threads_started":1}
W20260619 18:51:21.253206 20727 leader_election.cc:341] T 100545edd4ac4748b63f1f782249c369 P 4c5611deb12845d982674dcfea33fd93 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bad6f9f3febb40e8b9161ba802d06542 (127.3.243.67:40449): Not found: Tablet not found: 100545edd4ac4748b63f1f782249c369
W20260619 18:51:21.253497 20728 leader_election.cc:341] T 100545edd4ac4748b63f1f782249c369 P 4c5611deb12845d982674dcfea33fd93 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2ab38de74582419e893b702c9dd7a051 (127.3.243.66:37273): Not found: Tablet not found: 100545edd4ac4748b63f1f782249c369
I20260619 18:51:21.253638 20728 leader_election.cc:302] T 100545edd4ac4748b63f1f782249c369 P 4c5611deb12845d982674dcfea33fd93 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c5611deb12845d982674dcfea33fd93; no voters: 2ab38de74582419e893b702c9dd7a051, bad6f9f3febb40e8b9161ba802d06542
I20260619 18:51:21.253998 21108 raft_consensus.cc:2726] T 100545edd4ac4748b63f1f782249c369 P 4c5611deb12845d982674dcfea33fd93 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 18:51:21.523365 21113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:21.522480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:42879 (local address 127.3.243.67:53419)
0619 18:51:21.522941 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:21.523075 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:42879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":203,"threads_started":1}
I20260619 18:51:21.670189 21109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:21.669821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:42879 (local address 127.3.243.65:46065)
0619 18:51:21.669971 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:21.670058 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:42879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260619 18:51:22.089130 21114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:22.088504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:42879 (local address 127.3.243.66:40563)
0619 18:51:22.088898 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:22.088975 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:42879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":207,"threads_started":1}
I20260619 18:51:22.524770 21115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:22.524144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:42879 (local address 127.3.243.67:35139)
0619 18:51:22.524549 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:22.524624 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:42879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":195,"threads_started":1}
I20260619 18:51:22.671734 21116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:22.670959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:42879 (local address 127.3.243.65:57171)
0619 18:51:22.671454 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:22.671556 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:42879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":280,"threads_started":1}
I20260619 18:51:23.090492 21117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:23.089861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:42879 (local address 127.3.243.66:47997)
0619 18:51:23.090259 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:23.090333 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:42879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":210,"threads_started":1}
I20260619 18:51:23.502549 21118 raft_consensus.cc:482] T 100545edd4ac4748b63f1f782249c369 P 4c5611deb12845d982674dcfea33fd93 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:51:23.502728 21118 raft_consensus.cc:504] T 100545edd4ac4748b63f1f782249c369 P 4c5611deb12845d982674dcfea33fd93 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bad6f9f3febb40e8b9161ba802d06542" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40449 } } peers { permanent_uuid: "4c5611deb12845d982674dcfea33fd93" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 38277 } } peers { permanent_uuid: "2ab38de74582419e893b702c9dd7a051" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 37273 } }
I20260619 18:51:23.503316 21118 leader_election.cc:288] T 100545edd4ac4748b63f1f782249c369 P 4c5611deb12845d982674dcfea33fd93 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bad6f9f3febb40e8b9161ba802d06542 (127.3.243.67:40449), 2ab38de74582419e893b702c9dd7a051 (127.3.243.66:37273)
W20260619 18:51:23.504331 20727 leader_election.cc:341] T 100545edd4ac4748b63f1f782249c369 P 4c5611deb12845d982674dcfea33fd93 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bad6f9f3febb40e8b9161ba802d06542 (127.3.243.67:40449): Not found: Tablet not found: 100545edd4ac4748b63f1f782249c369
W20260619 18:51:23.504515 20728 leader_election.cc:341] T 100545edd4ac4748b63f1f782249c369 P 4c5611deb12845d982674dcfea33fd93 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2ab38de74582419e893b702c9dd7a051 (127.3.243.66:37273): Not found: Tablet not found: 100545edd4ac4748b63f1f782249c369
I20260619 18:51:23.504630 20728 leader_election.cc:302] T 100545edd4ac4748b63f1f782249c369 P 4c5611deb12845d982674dcfea33fd93 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c5611deb12845d982674dcfea33fd93; no voters: 2ab38de74582419e893b702c9dd7a051, bad6f9f3febb40e8b9161ba802d06542
I20260619 18:51:23.504860 21118 raft_consensus.cc:2726] T 100545edd4ac4748b63f1f782249c369 P 4c5611deb12845d982674dcfea33fd93 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 18:51:23.526187 21119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:23.525501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:42879 (local address 127.3.243.67:51847)
0619 18:51:23.525923 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:23.526022 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:42879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":228,"threads_started":1}
I20260619 18:51:23.673224 21120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:23.672475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:42879 (local address 127.3.243.65:45883)
0619 18:51:23.672927 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:23.673040 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:42879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":228,"threads_started":1}
I20260619 18:51:24.092090 21121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:24.091358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:42879 (local address 127.3.243.66:42189)
0619 18:51:24.091762 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:24.091865 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:42879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":193,"threads_started":1}
I20260619 18:51:24.527563 21122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:24.526921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:42879 (local address 127.3.243.67:36293)
0619 18:51:24.527317 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:24.527391 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:42879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":198,"threads_started":1}
I20260619 18:51:24.674913 21123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:24.674147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:42879 (local address 127.3.243.65:36619)
0619 18:51:24.674625 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:24.674706 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:42879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":220,"threads_started":1}
I20260619 18:51:25.093670 21124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:25.092987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:42879 (local address 127.3.243.66:46353)
0619 18:51:25.093417 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:25.093501 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:42879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":221,"threads_started":1}
I20260619 18:51:25.529014 21125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:25.528350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:42879 (local address 127.3.243.67:56605)
0619 18:51:25.528766 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:25.528855 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:42879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":228,"threads_started":1}
I20260619 18:51:25.676440 21126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:25.675696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:42879 (local address 127.3.243.65:41185)
0619 18:51:25.676119 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:25.676241 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:42879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":228,"threads_started":1}
I20260619 18:51:26.095124 21127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:26.094425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:42879 (local address 127.3.243.66:38607)
0619 18:51:26.094862 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:26.094938 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:42879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":241,"threads_started":1}
I20260619 18:51:26.128898 21128 raft_consensus.cc:482] T 100545edd4ac4748b63f1f782249c369 P 4c5611deb12845d982674dcfea33fd93 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:51:26.129118 21128 raft_consensus.cc:504] T 100545edd4ac4748b63f1f782249c369 P 4c5611deb12845d982674dcfea33fd93 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bad6f9f3febb40e8b9161ba802d06542" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40449 } } peers { permanent_uuid: "4c5611deb12845d982674dcfea33fd93" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 38277 } } peers { permanent_uuid: "2ab38de74582419e893b702c9dd7a051" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 37273 } }
I20260619 18:51:26.129778 21128 leader_election.cc:288] T 100545edd4ac4748b63f1f782249c369 P 4c5611deb12845d982674dcfea33fd93 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bad6f9f3febb40e8b9161ba802d06542 (127.3.243.67:40449), 2ab38de74582419e893b702c9dd7a051 (127.3.243.66:37273)
W20260619 18:51:26.130584 20727 leader_election.cc:341] T 100545edd4ac4748b63f1f782249c369 P 4c5611deb12845d982674dcfea33fd93 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bad6f9f3febb40e8b9161ba802d06542 (127.3.243.67:40449): Not found: Tablet not found: 100545edd4ac4748b63f1f782249c369
W20260619 18:51:26.130753 20728 leader_election.cc:341] T 100545edd4ac4748b63f1f782249c369 P 4c5611deb12845d982674dcfea33fd93 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2ab38de74582419e893b702c9dd7a051 (127.3.243.66:37273): Not found: Tablet not found: 100545edd4ac4748b63f1f782249c369
I20260619 18:51:26.130846 20728 leader_election.cc:302] T 100545edd4ac4748b63f1f782249c369 P 4c5611deb12845d982674dcfea33fd93 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c5611deb12845d982674dcfea33fd93; no voters: 2ab38de74582419e893b702c9dd7a051, bad6f9f3febb40e8b9161ba802d06542
I20260619 18:51:26.131040 21128 raft_consensus.cc:2726] T 100545edd4ac4748b63f1f782249c369 P 4c5611deb12845d982674dcfea33fd93 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 18:51:26.530450 21129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:26.529800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:42879 (local address 127.3.243.67:50379)
0619 18:51:26.530205 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:26.530287 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:42879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":211,"threads_started":1}
I20260619 18:51:26.677883 21130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:26.677204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:42879 (local address 127.3.243.65:52315)
0619 18:51:26.677609 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:26.677693 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:42879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":224,"threads_started":1}
I20260619 18:51:27.096698 21131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:27.095937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:42879 (local address 127.3.243.66:42679)
0619 18:51:27.096423 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:27.096523 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:42879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":212,"threads_started":1}
I20260619 18:51:27.532166 21132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:27.531378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:42879 (local address 127.3.243.67:60149)
0619 18:51:27.531882 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:27.531967 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:42879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":219,"threads_started":1}
I20260619 18:51:27.645979 21133 raft_consensus.cc:482] T 100545edd4ac4748b63f1f782249c369 P 4c5611deb12845d982674dcfea33fd93 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:51:27.646158 21133 raft_consensus.cc:504] T 100545edd4ac4748b63f1f782249c369 P 4c5611deb12845d982674dcfea33fd93 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bad6f9f3febb40e8b9161ba802d06542" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40449 } } peers { permanent_uuid: "4c5611deb12845d982674dcfea33fd93" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 38277 } } peers { permanent_uuid: "2ab38de74582419e893b702c9dd7a051" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 37273 } }
I20260619 18:51:27.646723 21133 leader_election.cc:288] T 100545edd4ac4748b63f1f782249c369 P 4c5611deb12845d982674dcfea33fd93 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bad6f9f3febb40e8b9161ba802d06542 (127.3.243.67:40449), 2ab38de74582419e893b702c9dd7a051 (127.3.243.66:37273)
W20260619 18:51:27.647495 20727 leader_election.cc:341] T 100545edd4ac4748b63f1f782249c369 P 4c5611deb12845d982674dcfea33fd93 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bad6f9f3febb40e8b9161ba802d06542 (127.3.243.67:40449): Not found: Tablet not found: 100545edd4ac4748b63f1f782249c369
W20260619 18:51:27.647655 20728 leader_election.cc:341] T 100545edd4ac4748b63f1f782249c369 P 4c5611deb12845d982674dcfea33fd93 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2ab38de74582419e893b702c9dd7a051 (127.3.243.66:37273): Not found: Tablet not found: 100545edd4ac4748b63f1f782249c369
I20260619 18:51:27.647781 20728 leader_election.cc:302] T 100545edd4ac4748b63f1f782249c369 P 4c5611deb12845d982674dcfea33fd93 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c5611deb12845d982674dcfea33fd93; no voters: 2ab38de74582419e893b702c9dd7a051, bad6f9f3febb40e8b9161ba802d06542
I20260619 18:51:27.648005 21133 raft_consensus.cc:2726] T 100545edd4ac4748b63f1f782249c369 P 4c5611deb12845d982674dcfea33fd93 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 18:51:27.679378 21134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:27.678630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:42879 (local address 127.3.243.65:56973)
0619 18:51:27.679077 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:27.679184 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:42879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":239,"threads_started":1}
I20260619 18:51:28.098199 21135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:28.097451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:42879 (local address 127.3.243.66:48021)
0619 18:51:28.097907 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:28.098021 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:42879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":221,"threads_started":1}
I20260619 18:51:28.533620 21136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:28.532921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:42879 (local address 127.3.243.67:40971)
0619 18:51:28.533330 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:28.533416 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:42879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":184,"threads_started":1}
I20260619 18:51:28.680851 21137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:28.680123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:42879 (local address 127.3.243.65:35047)
0619 18:51:28.680575 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:28.680658 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:42879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":220,"threads_started":1}
I20260619 18:51:29.099723 21138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:29.099017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:42879 (local address 127.3.243.66:50271)
0619 18:51:29.099485 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:29.099572 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:42879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":236,"threads_started":1}
I20260619 18:51:29.535187 21139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:29.534486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:42879 (local address 127.3.243.67:38651)
0619 18:51:29.534943 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:29.535035 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:42879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":214,"threads_started":1}
I20260619 18:51:29.682389 21140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:29.681687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:42879 (local address 127.3.243.65:51801)
0619 18:51:29.682117 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:29.682196 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:42879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":234,"threads_started":1}
I20260619 18:51:30.101298 21141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:30.100549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:42879 (local address 127.3.243.66:57849)
0619 18:51:30.100991 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:30.101076 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:42879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":225,"threads_started":1}
I20260619 18:51:30.536661 21142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:30.535967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:42879 (local address 127.3.243.67:56939)
0619 18:51:30.536405 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:30.536492 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:42879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":243,"threads_started":1}
I20260619 18:51:30.683876 21143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:30.683217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:42879 (local address 127.3.243.65:46487)
0619 18:51:30.683622 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:30.683709 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:42879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":216,"threads_started":1}
I20260619 18:51:31.102785 21144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:31.102106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:42879 (local address 127.3.243.66:48381)
0619 18:51:31.102533 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:31.102620 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:42879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":214,"threads_started":1}
I20260619 18:51:31.538162 21145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:31.537466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:42879 (local address 127.3.243.67:55107)
0619 18:51:31.537926 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:31.538010 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:42879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":225,"threads_started":1}
I20260619 18:51:31.685406 21146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:31.684689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:42879 (local address 127.3.243.65:49845)
0619 18:51:31.685135 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:31.685214 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:42879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":242,"threads_started":1}
I20260619 18:51:32.104384 21147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:32.103615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:42879 (local address 127.3.243.66:47297)
0619 18:51:32.104065 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:32.104181 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:42879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":235,"threads_started":1}
I20260619 18:51:32.539629 21148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:32.538953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:42879 (local address 127.3.243.67:51251)
0619 18:51:32.539371 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:32.539465 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:42879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":204,"threads_started":1}
I20260619 18:51:32.686815 21149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:32.686140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:42879 (local address 127.3.243.65:46495)
0619 18:51:32.686556 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:32.686636 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:42879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":204,"threads_started":1}
I20260619 18:51:33.105942 21150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:33.105239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:42879 (local address 127.3.243.66:60033)
0619 18:51:33.105681 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:33.105771 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:42879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":215,"threads_started":1}
I20260619 18:51:33.541163 21151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:33.540429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:42879 (local address 127.3.243.67:48925)
0619 18:51:33.540904 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:33.540992 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:42879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":250,"threads_started":1}
I20260619 18:51:33.688326 21152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:33.687607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:42879 (local address 127.3.243.65:56811)
0619 18:51:33.688016 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:33.688100 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:42879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":220,"threads_started":1}
I20260619 18:51:34.107467 21153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:34.106727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:42879 (local address 127.3.243.66:55869)
0619 18:51:34.107168 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:34.107249 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:42879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":208,"threads_started":1}
I20260619 18:51:34.462819 21154 raft_consensus.cc:482] T 100545edd4ac4748b63f1f782249c369 P 4c5611deb12845d982674dcfea33fd93 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:51:34.463006 21154 raft_consensus.cc:504] T 100545edd4ac4748b63f1f782249c369 P 4c5611deb12845d982674dcfea33fd93 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bad6f9f3febb40e8b9161ba802d06542" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40449 } } peers { permanent_uuid: "4c5611deb12845d982674dcfea33fd93" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 38277 } } peers { permanent_uuid: "2ab38de74582419e893b702c9dd7a051" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 37273 } }
I20260619 18:51:34.463649 21154 leader_election.cc:288] T 100545edd4ac4748b63f1f782249c369 P 4c5611deb12845d982674dcfea33fd93 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bad6f9f3febb40e8b9161ba802d06542 (127.3.243.67:40449), 2ab38de74582419e893b702c9dd7a051 (127.3.243.66:37273)
W20260619 18:51:34.464565 20728 leader_election.cc:341] T 100545edd4ac4748b63f1f782249c369 P 4c5611deb12845d982674dcfea33fd93 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2ab38de74582419e893b702c9dd7a051 (127.3.243.66:37273): Not found: Tablet not found: 100545edd4ac4748b63f1f782249c369
W20260619 18:51:34.464726 20727 leader_election.cc:341] T 100545edd4ac4748b63f1f782249c369 P 4c5611deb12845d982674dcfea33fd93 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bad6f9f3febb40e8b9161ba802d06542 (127.3.243.67:40449): Not found: Tablet not found: 100545edd4ac4748b63f1f782249c369
I20260619 18:51:34.464859 20727 leader_election.cc:302] T 100545edd4ac4748b63f1f782249c369 P 4c5611deb12845d982674dcfea33fd93 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c5611deb12845d982674dcfea33fd93; no voters: 2ab38de74582419e893b702c9dd7a051, bad6f9f3febb40e8b9161ba802d06542
I20260619 18:51:34.465098 21154 raft_consensus.cc:2726] T 100545edd4ac4748b63f1f782249c369 P 4c5611deb12845d982674dcfea33fd93 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 18:51:34.542681 21155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:34.541978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:42879 (local address 127.3.243.67:33575)
0619 18:51:34.542393 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:34.542491 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:42879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":208,"threads_started":1}
I20260619 18:51:34.690263 21156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:34.689452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:42879 (local address 127.3.243.65:34481)
0619 18:51:34.689937 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:34.690050 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:42879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":230,"threads_started":1}
I20260619 18:51:35.109100 21157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:35.108439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:42879 (local address 127.3.243.66:52951)
0619 18:51:35.108826 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:35.108931 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:42879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":182,"threads_started":1}
I20260619 18:51:35.522403 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 20669
I20260619 18:51:35.536916 21158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:35.523311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37642 (local address 127.3.243.65:38277)
0619 18:51:35.524456 (+ 1145us) server_negotiation.cc:206] Beginning negotiation
0619 18:51:35.524466 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:51:35.524499 (+ 33us) server_negotiation.cc:407] Connection header received
0619 18:51:35.524572 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:51:35.524582 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:51:35.524872 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:51:35.525043 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:51:35.526705 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:35.527696 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:35.528885 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:35.529436 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:35.529528 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:35.531884 (+ 2356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:51:35.531895 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:51:35.531908 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:51:35.531943 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:51:35.534716 (+ 2773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:51:35.535311 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:51:35.535318 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:51:35.535320 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:51:35.535444 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:51:35.535723 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:51:35.535726 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:51:35.535728 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:51:35.536043 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:51:35.536321 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0619 18:51:35.536491 (+ 170us) server_negotiation.cc:299] Negotiation successful
0619 18:51:35.536666 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":955,"thread_start_us":836,"threads_started":1}
I20260619 18:51:35.538091 20835 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:51:35.544404 21160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:35.543687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:42879 (local address 127.3.243.67:50197)
0619 18:51:35.544134 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:35.544269 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.126:42879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":235,"threads_started":1}
I20260619 18:51:35.624009 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 20669
I20260619 18:51:35.638422 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 20843
I20260619 18:51:35.650691 21161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:35.639070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56660 (local address 127.3.243.66:37273)
0619 18:51:35.639530 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0619 18:51:35.639534 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:51:35.639550 (+ 16us) server_negotiation.cc:407] Connection header received
0619 18:51:35.639604 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:51:35.639607 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:51:35.639670 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:51:35.639760 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:51:35.641155 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:35.641956 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:35.643172 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:35.643606 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:35.643699 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:35.646250 (+ 2551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:51:35.646254 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:51:35.646258 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:51:35.646291 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:51:35.648644 (+ 2353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:51:35.649236 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:51:35.649239 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:51:35.649241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:51:35.649362 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:51:35.649626 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:51:35.649629 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:51:35.649631 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:51:35.649881 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:51:35.650069 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0619 18:51:35.650247 (+ 178us) server_negotiation.cc:299] Negotiation successful
0619 18:51:35.650410 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":242,"threads_started":1}
I20260619 18:51:35.651338 20966 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:51:35.731904 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 20843
I20260619 18:51:35.745990 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 20974
I20260619 18:51:35.758873 21163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:35.746800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33592 (local address 127.3.243.67:40449)
0619 18:51:35.747352 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0619 18:51:35.747357 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:51:35.747373 (+ 16us) server_negotiation.cc:407] Connection header received
0619 18:51:35.747424 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:51:35.747427 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:51:35.747492 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:51:35.747590 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:51:35.748991 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:35.749755 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:35.750949 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:35.751353 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:35.751459 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:35.753927 (+ 2468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:51:35.753932 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:51:35.753936 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:51:35.753967 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:51:35.756466 (+ 2499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:51:35.757053 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:51:35.757058 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:51:35.757060 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:51:35.757195 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:51:35.757533 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:51:35.757537 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:51:35.757539 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:51:35.757788 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:51:35.758033 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0619 18:51:35.758314 (+ 281us) server_negotiation.cc:299] Negotiation successful
0619 18:51:35.758519 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":281,"threads_started":1}
I20260619 18:51:35.759490 21097 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:51:35.839807 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 20974
2026-06-19T18:51:35Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (63642 ms)
[----------] 30 tests from SecurityITest (285088 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894810715282-4045-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:51:35 dist-test-slave-m9wg krb5kdc[21167](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:51:35 dist-test-slave-m9wg krb5kdc[21167](info): set up 2 sockets
Jun 19 18:51:35 dist-test-slave-m9wg krb5kdc[21167](info): commencing operation
krb5kdc: starting...
W20260619 18:51:36.840191 4045 mini_kdc.cc:121] Time spent starting KDC: real 0.965s user 0.001s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894810715282-4045-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894810715282-4045-0/krb5kdc/test-user.keytab.
Jun 19 18:51:36 dist-test-slave-m9wg krb5kdc[21167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895096, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:51:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:51:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.243.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab.
Entry for principal kudu/127.3.243.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab.
I20260619 18:51:37.047139 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894810715282-4045-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894810715282-4045-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894810715282-4045-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.126:42681
--webserver_interface=127.3.243.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab
--principal=kudu/127.3.243.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:37135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.126:42681
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:51:37.351269 21180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:51:37.351534 21180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:51:37.351667 21180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:51:37.358693 21180 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:51:37.358819 21180 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:51:37.358896 21180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:51:37.358971 21180 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:51:37.359030 21180 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:51:37.368537 21180 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:37135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894810715282-4045-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894810715282-4045-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.243.126:42681
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab
--principal=kudu/127.3.243.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.126:42681
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.243.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894810715282-4045-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:51:37.369745 21180 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:51:37.371027 21180 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:51:37.381920 21185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:51:37.381852 21186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:51:37.381894 21188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:51:37.382445 21180 server_base.cc:1034] running on GCE node
I20260619 18:51:37.383322 21180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:51:37.384655 21180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:51:37.385887 21180 hybrid_clock.cc:648] HybridClock initialized: now 1781895097385848 us; error 49 us; skew 500 ppm
I20260619 18:51:37.386224 21180 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:51:37 dist-test-slave-m9wg krb5kdc[21167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895097, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:51:37.393821 21180 init.cc:377] Logged in from keytab as kudu/127.3.243.126@KRBTEST.COM (short username kudu)
I20260619 18:51:37.395197 21180 webserver.cc:460] Webserver started at http://127.3.243.126:40027/ using document root <none> and password file <none>
I20260619 18:51:37.395836 21180 fs_manager.cc:359] Metadata directory not provided
I20260619 18:51:37.395951 21180 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:51:37.396278 21180 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:51:37.399103 21180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894810715282-4045-0/minicluster-data/master-0/data/instance:
uuid: "24f8c13db7404fe0a7efd4631402409a"
format_stamp: "Formatted at 2026-06-19 18:51:37 on dist-test-slave-m9wg"
server_key: "394fb91246fc4c20b76ff2820713c832"
server_key_iv: "d14abf4b81c6356da78958e31d2d8972"
server_key_version: "encryptionkey@0"
I20260619 18:51:37.399813 21180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894810715282-4045-0/minicluster-data/master-0/wal/instance:
uuid: "24f8c13db7404fe0a7efd4631402409a"
format_stamp: "Formatted at 2026-06-19 18:51:37 on dist-test-slave-m9wg"
server_key: "394fb91246fc4c20b76ff2820713c832"
server_key_iv: "d14abf4b81c6356da78958e31d2d8972"
server_key_version: "encryptionkey@0"
I20260619 18:51:37.404582 21180 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.002s
I20260619 18:51:37.407675 21195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:51:37.408869 21180 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 18:51:37.409015 21180 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894810715282-4045-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894810715282-4045-0/minicluster-data/master-0/wal
uuid: "24f8c13db7404fe0a7efd4631402409a"
format_stamp: "Formatted at 2026-06-19 18:51:37 on dist-test-slave-m9wg"
server_key: "394fb91246fc4c20b76ff2820713c832"
server_key_iv: "d14abf4b81c6356da78958e31d2d8972"
server_key_version: "encryptionkey@0"
I20260619 18:51:37.409173 21180 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894810715282-4045-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894810715282-4045-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894810715282-4045-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:51:37.430033 21180 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:51:37.433507 21180 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:51:37.433740 21180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:51:37.450176 21180 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.126:42681
I20260619 18:51:37.450196 21246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.126:42681 every 8 connection(s)
I20260619 18:51:37.451416 21180 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
I20260619 18:51:37.454664 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 21180
I20260619 18:51:37.454856 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894810715282-4045-0/minicluster-data/master-0/wal/instance
I20260619 18:51:37.454872 21247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:51:37.455477 4045 external_mini_cluster.cc:1428] Setting key 136593386cd6660a9d45d8a82d39e218
I20260619 18:51:37.467339 21247 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24f8c13db7404fe0a7efd4631402409a: Bootstrap starting.
Jun 19 18:51:37 dist-test-slave-m9wg krb5kdc[21167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895096, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.126@KRBTEST.COM
I20260619 18:51:37.470815 21247 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 24f8c13db7404fe0a7efd4631402409a: Neither blocks nor log segments found. Creating new log.
I20260619 18:51:37.471874 21247 log.cc:824] T 00000000000000000000000000000000 P 24f8c13db7404fe0a7efd4631402409a: Log is configured to *not* fsync() on all Append() calls
I20260619 18:51:37.474936 21247 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24f8c13db7404fe0a7efd4631402409a: No bootstrap required, opened a new log
I20260619 18:51:37.479054 21250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:37.457312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50420 (local address 127.3.243.126:42681)
0619 18:51:37.458070 (+ 758us) server_negotiation.cc:206] Beginning negotiation
0619 18:51:37.458082 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 18:51:37.458117 (+ 35us) server_negotiation.cc:407] Connection header received
0619 18:51:37.460323 (+ 2206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:51:37.460344 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:51:37.460702 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:51:37.461033 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:51:37.462702 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:37.464101 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:37.465938 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:37.466606 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:37.466741 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:37.471369 (+ 4628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:51:37.471377 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:51:37.471390 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:51:37.471435 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:51:37.474905 (+ 3470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:51:37.475553 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:51:37.475561 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:51:37.475568 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:51:37.475760 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:51:37.476084 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:51:37.476088 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:51:37.476092 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:51:37.476743 (+ 651us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:51:37.477021 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0619 18:51:37.477344 (+ 323us) server_negotiation.cc:299] Negotiation successful
0619 18:51:37.477749 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":367,"threads_started":1}
I20260619 18:51:37.483222 21247 raft_consensus.cc:348] T 00000000000000000000000000000000 P 24f8c13db7404fe0a7efd4631402409a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24f8c13db7404fe0a7efd4631402409a" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 42681 } }
I20260619 18:51:37.483528 21247 raft_consensus.cc:374] T 00000000000000000000000000000000 P 24f8c13db7404fe0a7efd4631402409a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:51:37.483595 21247 raft_consensus.cc:729] T 00000000000000000000000000000000 P 24f8c13db7404fe0a7efd4631402409a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24f8c13db7404fe0a7efd4631402409a, State: Initialized, Role: FOLLOWER
I20260619 18:51:37.484025 21247 consensus_queue.cc:260] T 00000000000000000000000000000000 P 24f8c13db7404fe0a7efd4631402409a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24f8c13db7404fe0a7efd4631402409a" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 42681 } }
I20260619 18:51:37.484241 21247 raft_consensus.cc:388] T 00000000000000000000000000000000 P 24f8c13db7404fe0a7efd4631402409a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:51:37.484336 21247 raft_consensus.cc:482] T 00000000000000000000000000000000 P 24f8c13db7404fe0a7efd4631402409a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:51:37.484453 21247 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 24f8c13db7404fe0a7efd4631402409a [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:51:37.486362 21247 raft_consensus.cc:504] T 00000000000000000000000000000000 P 24f8c13db7404fe0a7efd4631402409a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24f8c13db7404fe0a7efd4631402409a" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 42681 } }
I20260619 18:51:37.486757 21247 leader_election.cc:302] T 00000000000000000000000000000000 P 24f8c13db7404fe0a7efd4631402409a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 24f8c13db7404fe0a7efd4631402409a; no voters:
I20260619 18:51:37.487221 21247 leader_election.cc:288] T 00000000000000000000000000000000 P 24f8c13db7404fe0a7efd4631402409a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:51:37.487323 21252 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 24f8c13db7404fe0a7efd4631402409a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:51:37.487906 21252 raft_consensus.cc:686] T 00000000000000000000000000000000 P 24f8c13db7404fe0a7efd4631402409a [term 1 LEADER]: Becoming Leader. State: Replica: 24f8c13db7404fe0a7efd4631402409a, State: Running, Role: LEADER
I20260619 18:51:37.488482 21252 consensus_queue.cc:237] T 00000000000000000000000000000000 P 24f8c13db7404fe0a7efd4631402409a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24f8c13db7404fe0a7efd4631402409a" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 42681 } }
I20260619 18:51:37.488974 21247 sys_catalog.cc:564] T 00000000000000000000000000000000 P 24f8c13db7404fe0a7efd4631402409a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:51:37.490758 21253 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24f8c13db7404fe0a7efd4631402409a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "24f8c13db7404fe0a7efd4631402409a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24f8c13db7404fe0a7efd4631402409a" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 42681 } } }
I20260619 18:51:37.491088 21253 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24f8c13db7404fe0a7efd4631402409a [sys.catalog]: This master's current role is: LEADER
I20260619 18:51:37.491703 21254 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24f8c13db7404fe0a7efd4631402409a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24f8c13db7404fe0a7efd4631402409a. Latest consensus state: current_term: 1 leader_uuid: "24f8c13db7404fe0a7efd4631402409a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24f8c13db7404fe0a7efd4631402409a" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 42681 } } }
I20260619 18:51:37.492094 21254 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24f8c13db7404fe0a7efd4631402409a [sys.catalog]: This master's current role is: LEADER
I20260619 18:51:37.495445 21257 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:51:37.500809 21257 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:51:37.509564 21257 catalog_manager.cc:1347] Generated new cluster ID: bd80b21593d648e694bd01a5201acb3f
I20260619 18:51:37.509654 21257 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:51:37.526424 21257 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:51:37.527869 21257 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:51:37.536912 21257 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 24f8c13db7404fe0a7efd4631402409a: Generated new TSK 0
I20260619 18:51:37.537853 21257 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.243.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.65.keytab.
Entry for principal kudu/127.3.243.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.65.keytab.
I20260619 18:51:37.595381 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894810715282-4045-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894810715282-4045-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894810715282-4045-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894810715282-4045-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.65:0
--local_ip_for_outbound_sockets=127.3.243.65
--webserver_interface=127.3.243.65
--webserver_port=0
--tserver_master_addrs=127.3.243.126:42681
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.65.keytab
--principal=kudu/127.3.243.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:37135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:51:37.899295 21273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:51:37.899567 21273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:51:37.899668 21273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:51:37.906719 21273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:51:37.906819 21273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:51:37.906930 21273 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.65
I20260619 18:51:37.916673 21273 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:37135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894810715282-4045-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894810715282-4045-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.65.keytab
--principal=kudu/127.3.243.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894810715282-4045-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.243.65
--webserver_port=0
--tserver_master_addrs=127.3.243.126:42681
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.65
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894810715282-4045-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:51:37.917807 21273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:51:37.918989 21273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:51:37.930923 21278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:51:37.931102 21279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:51:37.931293 21281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:51:37.931679 21273 server_base.cc:1034] running on GCE node
I20260619 18:51:37.932303 21273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:51:37.933127 21273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:51:37.934310 21273 hybrid_clock.cc:648] HybridClock initialized: now 1781895097934266 us; error 36 us; skew 500 ppm
I20260619 18:51:37.934583 21273 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:51:37 dist-test-slave-m9wg krb5kdc[21167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895097, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:51:37.937542 21273 init.cc:377] Logged in from keytab as kudu/127.3.243.65@KRBTEST.COM (short username kudu)
I20260619 18:51:37.938862 21273 webserver.cc:460] Webserver started at http://127.3.243.65:45575/ using document root <none> and password file <none>
I20260619 18:51:37.939472 21273 fs_manager.cc:359] Metadata directory not provided
I20260619 18:51:37.939574 21273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:51:37.939847 21273 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:51:37.942517 21273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894810715282-4045-0/minicluster-data/ts-0/data/instance:
uuid: "0d970abe09234fbb98ad3965c3bc99d7"
format_stamp: "Formatted at 2026-06-19 18:51:37 on dist-test-slave-m9wg"
server_key: "99cff02e0396fd90c31003985204c8fd"
server_key_iv: "d3e025b58aaf50c5fa2875d89e113dab"
server_key_version: "encryptionkey@0"
I20260619 18:51:37.943279 21273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894810715282-4045-0/minicluster-data/ts-0/wal/instance:
uuid: "0d970abe09234fbb98ad3965c3bc99d7"
format_stamp: "Formatted at 2026-06-19 18:51:37 on dist-test-slave-m9wg"
server_key: "99cff02e0396fd90c31003985204c8fd"
server_key_iv: "d3e025b58aaf50c5fa2875d89e113dab"
server_key_version: "encryptionkey@0"
I20260619 18:51:37.947892 21273 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260619 18:51:37.951030 21288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:51:37.952226 21273 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 18:51:37.952361 21273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894810715282-4045-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894810715282-4045-0/minicluster-data/ts-0/wal
uuid: "0d970abe09234fbb98ad3965c3bc99d7"
format_stamp: "Formatted at 2026-06-19 18:51:37 on dist-test-slave-m9wg"
server_key: "99cff02e0396fd90c31003985204c8fd"
server_key_iv: "d3e025b58aaf50c5fa2875d89e113dab"
server_key_version: "encryptionkey@0"
I20260619 18:51:37.952526 21273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894810715282-4045-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894810715282-4045-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894810715282-4045-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:51:37.966140 21273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:51:37.969499 21273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:51:37.969735 21273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:51:37.970551 21273 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:51:37.971704 21273 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:51:37.971777 21273 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:51:37.971891 21273 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:51:37.971947 21273 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:51:38.000901 21273 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.65:42687
I20260619 18:51:38.000944 21400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.65:42687 every 8 connection(s)
I20260619 18:51:38.002116 21273 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894810715282-4045-0/minicluster-data/ts-0/data/info.pb
I20260619 18:51:38.004918 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 21273
I20260619 18:51:38.005126 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894810715282-4045-0/minicluster-data/ts-0/wal/instance
I20260619 18:51:38.005789 4045 external_mini_cluster.cc:1428] Setting key b3e5da0429bcd7bae93a29b2782ee2d7
Jun 19 18:51:38 dist-test-slave-m9wg krb5kdc[21167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895097, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.65@KRBTEST.COM for kudu/127.3.243.126@KRBTEST.COM
I20260619 18:51:38.022814 21403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:38.004599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.65:49839 (local address 127.3.243.126:42681)
0619 18:51:38.005115 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0619 18:51:38.005122 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:51:38.006124 (+ 1002us) server_negotiation.cc:407] Connection header received
0619 18:51:38.008073 (+ 1949us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:51:38.008079 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:51:38.008181 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:51:38.008302 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:51:38.009688 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:38.010633 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:38.012722 (+ 2089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:38.013314 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:38.013435 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:38.016957 (+ 3522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:51:38.016964 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:51:38.016969 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:51:38.017011 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:51:38.019846 (+ 2835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:51:38.020559 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:51:38.020565 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:51:38.020569 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:51:38.020695 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:51:38.021167 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:51:38.021171 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:51:38.021174 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:51:38.021512 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:51:38.021645 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 18:51:38.022257 (+ 612us) server_negotiation.cc:299] Negotiation successful
0619 18:51:38.022470 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":246,"threads_started":1}
I20260619 18:51:38.023586 21404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:38.005194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:42681 (local address 127.3.243.65:49839)
0619 18:51:38.005945 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:38.005999 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 18:51:38.007819 (+ 1820us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:51:38.008482 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:51:38.008489 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:51:38.008968 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:51:38.009520 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:51:38.009538 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:51:38.010826 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:51:38.010832 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:51:38.012556 (+ 1724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:51:38.012571 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:51:38.013641 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:51:38.013648 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:51:38.013848 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:38.014468 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:51:38.014490 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:51:38.016740 (+ 2250us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:51:38.019988 (+ 3248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:51:38.019995 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:51:38.020006 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:51:38.020461 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:51:38.020828 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:51:38.020832 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:51:38.020835 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:51:38.021054 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:51:38.021613 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:51:38.021618 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:51:38.021931 (+ 313us) client_negotiation.cc:769] Sending connection context
0619 18:51:38.022145 (+ 214us) client_negotiation.cc:240] Negotiation successful
0619 18:51:38.022514 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":290,"threads_started":1}
I20260619 18:51:38.025089 21401 heartbeater.cc:347] Connected to a master server at 127.3.243.126:42681
I20260619 18:51:38.025393 21401 heartbeater.cc:464] Registering TS with master...
I20260619 18:51:38.026096 21401 heartbeater.cc:511] Master 127.3.243.126:42681 requested a full tablet report, sending...
I20260619 18:51:38.028203 21212 ts_manager.cc:194] Registered new tserver with Master: 0d970abe09234fbb98ad3965c3bc99d7 (127.3.243.65:42687)
I20260619 18:51:38.029652 21212 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.65@KRBTEST.COM'} at 127.3.243.65:49839
WARNING: no policy specified for kudu/127.3.243.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.66.keytab.
Entry for principal kudu/127.3.243.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.66.keytab.
I20260619 18:51:38.056733 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894810715282-4045-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894810715282-4045-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894810715282-4045-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894810715282-4045-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.66:0
--local_ip_for_outbound_sockets=127.3.243.66
--webserver_interface=127.3.243.66
--webserver_port=0
--tserver_master_addrs=127.3.243.126:42681
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.66.keytab
--principal=kudu/127.3.243.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:37135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:51:38.355391 21407 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:51:38.355660 21407 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:51:38.355798 21407 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:51:38.362820 21407 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:51:38.362938 21407 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:51:38.363088 21407 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.66
I20260619 18:51:38.372258 21407 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:37135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894810715282-4045-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894810715282-4045-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.66.keytab
--principal=kudu/127.3.243.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894810715282-4045-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.243.66
--webserver_port=0
--tserver_master_addrs=127.3.243.126:42681
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.66
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894810715282-4045-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:51:38.373411 21407 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:51:38.374634 21407 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:51:38.386214 21415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:51:38.386260 21412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:51:38.386277 21413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:51:38.386880 21407 server_base.cc:1034] running on GCE node
I20260619 18:51:38.387388 21407 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:51:38.388211 21407 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:51:38.389420 21407 hybrid_clock.cc:648] HybridClock initialized: now 1781895098389364 us; error 43 us; skew 500 ppm
I20260619 18:51:38.389703 21407 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:51:38 dist-test-slave-m9wg krb5kdc[21167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895098, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:51:38.392583 21407 init.cc:377] Logged in from keytab as kudu/127.3.243.66@KRBTEST.COM (short username kudu)
I20260619 18:51:38.393841 21407 webserver.cc:460] Webserver started at http://127.3.243.66:33853/ using document root <none> and password file <none>
I20260619 18:51:38.394491 21407 fs_manager.cc:359] Metadata directory not provided
I20260619 18:51:38.394604 21407 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:51:38.394883 21407 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:51:38.397789 21407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894810715282-4045-0/minicluster-data/ts-1/data/instance:
uuid: "4c7959d1a2f54fd3850cd7736a5085f9"
format_stamp: "Formatted at 2026-06-19 18:51:38 on dist-test-slave-m9wg"
server_key: "91aa3500e49253e9cc633cf46cf45f93"
server_key_iv: "0d4d365b3e95b374789de65ca7d10030"
server_key_version: "encryptionkey@0"
I20260619 18:51:38.398552 21407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894810715282-4045-0/minicluster-data/ts-1/wal/instance:
uuid: "4c7959d1a2f54fd3850cd7736a5085f9"
format_stamp: "Formatted at 2026-06-19 18:51:38 on dist-test-slave-m9wg"
server_key: "91aa3500e49253e9cc633cf46cf45f93"
server_key_iv: "0d4d365b3e95b374789de65ca7d10030"
server_key_version: "encryptionkey@0"
I20260619 18:51:38.403239 21407 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260619 18:51:38.406384 21422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:51:38.407577 21407 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 18:51:38.407718 21407 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894810715282-4045-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894810715282-4045-0/minicluster-data/ts-1/wal
uuid: "4c7959d1a2f54fd3850cd7736a5085f9"
format_stamp: "Formatted at 2026-06-19 18:51:38 on dist-test-slave-m9wg"
server_key: "91aa3500e49253e9cc633cf46cf45f93"
server_key_iv: "0d4d365b3e95b374789de65ca7d10030"
server_key_version: "encryptionkey@0"
I20260619 18:51:38.407883 21407 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894810715282-4045-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894810715282-4045-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894810715282-4045-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:51:38.430050 21407 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:51:38.434016 21407 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:51:38.434291 21407 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:51:38.435168 21407 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:51:38.436383 21407 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:51:38.436456 21407 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:51:38.436571 21407 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:51:38.436623 21407 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:51:38.466862 21407 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.66:45187
I20260619 18:51:38.466933 21534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.66:45187 every 8 connection(s)
I20260619 18:51:38.468030 21407 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894810715282-4045-0/minicluster-data/ts-1/data/info.pb
I20260619 18:51:38.476263 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 21407
I20260619 18:51:38.476428 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894810715282-4045-0/minicluster-data/ts-1/wal/instance
I20260619 18:51:38.476835 4045 external_mini_cluster.cc:1428] Setting key bb801f2aceb879c3e64916de46de75b9
Jun 19 18:51:38 dist-test-slave-m9wg krb5kdc[21167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895098, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.66@KRBTEST.COM for kudu/127.3.243.126@KRBTEST.COM
I20260619 18:51:38.486419 21403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:38.470362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.66:53449 (local address 127.3.243.126:42681)
0619 18:51:38.470521 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 18:51:38.470525 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:51:38.471571 (+ 1046us) server_negotiation.cc:407] Connection header received
0619 18:51:38.473588 (+ 2017us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:51:38.473594 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:51:38.473671 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:51:38.473756 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:51:38.475136 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:38.475825 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:38.477401 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:38.477831 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:38.477946 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:38.481359 (+ 3413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:51:38.481364 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:51:38.481368 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:51:38.481400 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:51:38.483754 (+ 2354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:51:38.484504 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:51:38.484508 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:51:38.484510 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:51:38.484635 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:51:38.485040 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:51:38.485043 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:51:38.485045 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:51:38.485275 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:51:38.485374 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 18:51:38.486054 (+ 680us) server_negotiation.cc:299] Negotiation successful
0619 18:51:38.486207 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 18:51:38.487397 21537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:38.470771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:42681 (local address 127.3.243.66:53449)
0619 18:51:38.471429 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:38.471463 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 18:51:38.473341 (+ 1878us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:51:38.473981 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:51:38.473988 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:51:38.474477 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:51:38.474982 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:51:38.474998 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:51:38.475976 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:51:38.475979 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:51:38.477271 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:51:38.477282 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:51:38.478023 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:51:38.478031 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:51:38.478249 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:38.478824 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:51:38.478844 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:51:38.481162 (+ 2318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:51:38.483875 (+ 2713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:51:38.483883 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:51:38.483896 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:51:38.484408 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:51:38.484719 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:51:38.484722 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:51:38.484725 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:51:38.484952 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:51:38.485378 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:51:38.485385 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:51:38.485728 (+ 343us) client_negotiation.cc:769] Sending connection context
0619 18:51:38.485989 (+ 261us) client_negotiation.cc:240] Negotiation successful
0619 18:51:38.486424 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":246,"threads_started":1}
I20260619 18:51:38.488583 21535 heartbeater.cc:347] Connected to a master server at 127.3.243.126:42681
I20260619 18:51:38.488862 21535 heartbeater.cc:464] Registering TS with master...
I20260619 18:51:38.489615 21535 heartbeater.cc:511] Master 127.3.243.126:42681 requested a full tablet report, sending...
I20260619 18:51:38.490931 21212 ts_manager.cc:194] Registered new tserver with Master: 4c7959d1a2f54fd3850cd7736a5085f9 (127.3.243.66:45187)
I20260619 18:51:38.491688 21212 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.66@KRBTEST.COM'} at 127.3.243.66:53449
WARNING: no policy specified for kudu/127.3.243.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.67.keytab.
Entry for principal kudu/127.3.243.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.67.keytab.
I20260619 18:51:38.524935 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894810715282-4045-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894810715282-4045-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894810715282-4045-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894810715282-4045-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.67:0
--local_ip_for_outbound_sockets=127.3.243.67
--webserver_interface=127.3.243.67
--webserver_port=0
--tserver_master_addrs=127.3.243.126:42681
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.67.keytab
--principal=kudu/127.3.243.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:37135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:51:38.835371 21540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:51:38.835637 21540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:51:38.835769 21540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:51:38.842949 21540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:51:38.843050 21540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:51:38.843204 21540 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.67
I20260619 18:51:38.852813 21540 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:37135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894810715282-4045-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894810715282-4045-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.67.keytab
--principal=kudu/127.3.243.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894810715282-4045-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.243.67
--webserver_port=0
--tserver_master_addrs=127.3.243.126:42681
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.67
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894810715282-4045-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:51:38.854049 21540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:51:38.855324 21540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:51:38.867815 21546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:51:38.867921 21548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:51:38.867924 21545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:51:38.868736 21540 server_base.cc:1034] running on GCE node
I20260619 18:51:38.869299 21540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:51:38.870152 21540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:51:38.871369 21540 hybrid_clock.cc:648] HybridClock initialized: now 1781895098871304 us; error 55 us; skew 500 ppm
I20260619 18:51:38.871658 21540 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:51:38 dist-test-slave-m9wg krb5kdc[21167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895098, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:51:38.874715 21540 init.cc:377] Logged in from keytab as kudu/127.3.243.67@KRBTEST.COM (short username kudu)
I20260619 18:51:38.876121 21540 webserver.cc:460] Webserver started at http://127.3.243.67:44269/ using document root <none> and password file <none>
I20260619 18:51:38.876813 21540 fs_manager.cc:359] Metadata directory not provided
I20260619 18:51:38.876932 21540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:51:38.877278 21540 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:51:38.880282 21540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894810715282-4045-0/minicluster-data/ts-2/data/instance:
uuid: "d38dc8a8d46744ec8a4c69eadec67f03"
format_stamp: "Formatted at 2026-06-19 18:51:38 on dist-test-slave-m9wg"
server_key: "533f1ed605837eb251073a026dbc2a5c"
server_key_iv: "1bf4a62c78703ecf707e79a4f6699132"
server_key_version: "encryptionkey@0"
I20260619 18:51:38.881067 21540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894810715282-4045-0/minicluster-data/ts-2/wal/instance:
uuid: "d38dc8a8d46744ec8a4c69eadec67f03"
format_stamp: "Formatted at 2026-06-19 18:51:38 on dist-test-slave-m9wg"
server_key: "533f1ed605837eb251073a026dbc2a5c"
server_key_iv: "1bf4a62c78703ecf707e79a4f6699132"
server_key_version: "encryptionkey@0"
I20260619 18:51:38.886140 21540 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 18:51:38.889508 21555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:51:38.890798 21540 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 18:51:38.890935 21540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894810715282-4045-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894810715282-4045-0/minicluster-data/ts-2/wal
uuid: "d38dc8a8d46744ec8a4c69eadec67f03"
format_stamp: "Formatted at 2026-06-19 18:51:38 on dist-test-slave-m9wg"
server_key: "533f1ed605837eb251073a026dbc2a5c"
server_key_iv: "1bf4a62c78703ecf707e79a4f6699132"
server_key_version: "encryptionkey@0"
I20260619 18:51:38.891113 21540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894810715282-4045-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894810715282-4045-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894810715282-4045-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:51:38.924918 21540 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:51:38.929247 21540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:51:38.929515 21540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:51:38.930452 21540 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:51:38.931748 21540 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:51:38.931823 21540 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:51:38.931943 21540 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:51:38.931998 21540 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:51:38.961753 21540 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.67:33565
I20260619 18:51:38.961789 21667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.67:33565 every 8 connection(s)
I20260619 18:51:38.962978 21540 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894810715282-4045-0/minicluster-data/ts-2/data/info.pb
I20260619 18:51:38.964371 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 21540
I20260619 18:51:38.964555 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894810715282-4045-0/minicluster-data/ts-2/wal/instance
I20260619 18:51:38.964973 4045 external_mini_cluster.cc:1428] Setting key 791534fc2fa954987b2d102847960076
Jun 19 18:51:38 dist-test-slave-m9wg krb5kdc[21167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895098, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.67@KRBTEST.COM for kudu/127.3.243.126@KRBTEST.COM
I20260619 18:51:38.983127 21403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:38.967151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.67:49149 (local address 127.3.243.126:42681)
0619 18:51:38.967346 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 18:51:38.967352 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:51:38.968678 (+ 1326us) server_negotiation.cc:407] Connection header received
0619 18:51:38.970399 (+ 1721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:51:38.970403 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:51:38.970468 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:51:38.970544 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:51:38.971904 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:38.972665 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:38.974150 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:38.974570 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:38.974678 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:38.977827 (+ 3149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:51:38.977832 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:51:38.977835 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:51:38.977867 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:51:38.980749 (+ 2882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:51:38.981337 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:51:38.981341 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:51:38.981343 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:51:38.981447 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:51:38.981791 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:51:38.981795 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:51:38.981798 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:51:38.982068 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:51:38.982175 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 18:51:38.982720 (+ 545us) server_negotiation.cc:299] Negotiation successful
0619 18:51:38.982886 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 18:51:38.983969 21670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:38.967802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:42681 (local address 127.3.243.67:49149)
0619 18:51:38.968540 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:38.968578 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 18:51:38.970171 (+ 1593us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:51:38.970727 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:51:38.970734 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:51:38.971235 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:51:38.971763 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:51:38.971779 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:51:38.972786 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:51:38.972789 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:51:38.973996 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:51:38.974008 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:51:38.974784 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:51:38.974791 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:51:38.975022 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:38.975565 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:51:38.975590 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:51:38.977629 (+ 2039us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:51:38.980859 (+ 3230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:51:38.980866 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:51:38.980876 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:51:38.981257 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:51:38.981532 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:51:38.981535 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:51:38.981537 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:51:38.981698 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:51:38.982157 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:51:38.982162 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:51:38.982446 (+ 284us) client_negotiation.cc:769] Sending connection context
0619 18:51:38.982660 (+ 214us) client_negotiation.cc:240] Negotiation successful
0619 18:51:38.982997 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":322,"threads_started":1}
I20260619 18:51:38.985091 21668 heartbeater.cc:347] Connected to a master server at 127.3.243.126:42681
I20260619 18:51:38.985348 21668 heartbeater.cc:464] Registering TS with master...
I20260619 18:51:38.985998 21668 heartbeater.cc:511] Master 127.3.243.126:42681 requested a full tablet report, sending...
I20260619 18:51:38.987249 21212 ts_manager.cc:194] Registered new tserver with Master: d38dc8a8d46744ec8a4c69eadec67f03 (127.3.243.67:33565)
I20260619 18:51:38.988018 21212 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.67@KRBTEST.COM'} at 127.3.243.67:49149
I20260619 18:51:38.991993 4045 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 18:51:39.008671 21403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:38.996229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50432 (local address 127.3.243.126:42681)
0619 18:51:38.996357 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 18:51:38.996361 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:51:38.996845 (+ 484us) server_negotiation.cc:407] Connection header received
0619 18:51:38.997032 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:51:38.997035 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:51:38.997093 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:51:38.997182 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:51:38.998519 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:38.999218 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:39.000640 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:39.001274 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:39.001409 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:39.003584 (+ 2175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:51:39.003591 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:51:39.003596 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:51:39.003633 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:51:39.006604 (+ 2971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:51:39.007180 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:51:39.007183 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:51:39.007186 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:51:39.007301 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:51:39.007559 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:51:39.007561 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:51:39.007563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:51:39.007840 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:51:39.008027 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0619 18:51:39.008312 (+ 285us) server_negotiation.cc:299] Negotiation successful
0619 18:51:39.008466 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 18:51:39.012434 21212 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50432:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:51:39.019930 21212 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:51:39.032572 21401 heartbeater.cc:503] Master 127.3.243.126:42681 was elected leader, sending a full tablet report...
I20260619 18:51:39.052366 21683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:39.031810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39210 (local address 127.3.243.65:42687)
0619 18:51:39.042575 (+ 10765us) server_negotiation.cc:206] Beginning negotiation
0619 18:51:39.042585 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:51:39.043942 (+ 1357us) server_negotiation.cc:407] Connection header received
0619 18:51:39.044243 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:51:39.044251 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:51:39.044508 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:51:39.044693 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:51:39.046109 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:39.047679 (+ 1570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:39.050312 (+ 2633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:39.051274 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:39.051399 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:39.051635 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0619 18:51:39.051873 (+ 238us) server_negotiation.cc:299] Negotiation successful
0619 18:51:39.052045 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10567,"thread_start_us":10783,"threads_started":1}
I20260619 18:51:39.052452 21682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:39.031551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.65:42687 (local address 127.0.0.1:39210)
0619 18:51:39.043785 (+ 12234us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:39.043831 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 18:51:39.044043 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:51:39.044985 (+ 942us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:51:39.044994 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:51:39.045027 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:51:39.045530 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:51:39.045542 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:51:39.048239 (+ 2697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:51:39.048243 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:51:39.050161 (+ 1918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:51:39.050175 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:51:39.051479 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:51:39.051486 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:51:39.051674 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:39.051714 (+ 40us) client_negotiation.cc:769] Sending connection context
0619 18:51:39.051819 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 18:51:39.051991 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12080,"mutex_wait_us":25,"thread_start_us":11943,"threads_started":1}
I20260619 18:51:39.053146 21681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:39.031297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36458 (local address 127.3.243.67:33565)
0619 18:51:39.043312 (+ 12015us) server_negotiation.cc:206] Beginning negotiation
0619 18:51:39.043320 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:51:39.044217 (+ 897us) server_negotiation.cc:407] Connection header received
0619 18:51:39.044314 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:51:39.044321 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:51:39.044570 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:51:39.044760 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:51:39.046088 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:39.047689 (+ 1601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:39.050462 (+ 2773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:39.051818 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:39.051928 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:39.052137 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0619 18:51:39.052670 (+ 533us) server_negotiation.cc:299] Negotiation successful
0619 18:51:39.052826 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11835,"thread_start_us":11660,"threads_started":1}
I20260619 18:51:39.054106 21680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:39.031041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.67:33565 (local address 127.0.0.1:36458)
0619 18:51:39.043785 (+ 12744us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:39.043834 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 18:51:39.044015 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:51:39.045033 (+ 1018us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:51:39.045039 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:51:39.045097 (+ 58us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:51:39.045884 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:51:39.045894 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:51:39.047805 (+ 1911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:51:39.047810 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:51:39.050295 (+ 2485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:51:39.050309 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:51:39.052298 (+ 1989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:51:39.052304 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:51:39.052477 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:39.052507 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 18:51:39.053561 (+ 1054us) client_negotiation.cc:240] Negotiation successful
0619 18:51:39.053691 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12477,"mutex_wait_us":716,"spinlock_wait_cycles":12544,"thread_start_us":12310,"threads_started":1}
I20260619 18:51:39.054414 21684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:39.044418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.66:45187 (local address 127.0.0.1:54912)
0619 18:51:39.046192 (+ 1774us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:39.046206 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 18:51:39.046382 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:51:39.047149 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:51:39.047152 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:51:39.047171 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:51:39.047422 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:51:39.047429 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:51:39.049210 (+ 1781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:51:39.049216 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:51:39.051382 (+ 2166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:51:39.051397 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:51:39.053675 (+ 2278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:51:39.053681 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:51:39.053842 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:39.053883 (+ 41us) client_negotiation.cc:769] Sending connection context
0619 18:51:39.053977 (+ 94us) client_negotiation.cc:240] Negotiation successful
0619 18:51:39.054103 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1653,"thread_start_us":1523,"threads_started":1}
I20260619 18:51:39.054550 21685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:39.044477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54912 (local address 127.3.243.66:45187)
0619 18:51:39.045758 (+ 1281us) server_negotiation.cc:206] Beginning negotiation
0619 18:51:39.045768 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:51:39.046290 (+ 522us) server_negotiation.cc:407] Connection header received
0619 18:51:39.046570 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:51:39.046577 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:51:39.046773 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:51:39.046923 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:51:39.047565 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:39.049079 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:39.051568 (+ 2489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:39.053241 (+ 1673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:39.053356 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:39.053531 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0619 18:51:39.054050 (+ 519us) server_negotiation.cc:299] Negotiation successful
0619 18:51:39.054229 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1121,"thread_start_us":971,"threads_started":1}
I20260619 18:51:39.056680 21336 tablet_service.cc:1467] Processing CreateTablet for tablet 10557372aabd490f9b8340c475913aae (DEFAULT_TABLE table=test-table [id=45f1a2230ea94bff865c1fb10d2a5c9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:51:39.058164 21336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10557372aabd490f9b8340c475913aae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:51:39.057636 21603 tablet_service.cc:1467] Processing CreateTablet for tablet 10557372aabd490f9b8340c475913aae (DEFAULT_TABLE table=test-table [id=45f1a2230ea94bff865c1fb10d2a5c9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:51:39.058496 21470 tablet_service.cc:1467] Processing CreateTablet for tablet 10557372aabd490f9b8340c475913aae (DEFAULT_TABLE table=test-table [id=45f1a2230ea94bff865c1fb10d2a5c9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:51:39.059010 21603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10557372aabd490f9b8340c475913aae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:51:39.059772 21470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10557372aabd490f9b8340c475913aae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:51:39.070132 21686 tablet_bootstrap.cc:492] T 10557372aabd490f9b8340c475913aae P 0d970abe09234fbb98ad3965c3bc99d7: Bootstrap starting.
I20260619 18:51:39.073035 21687 tablet_bootstrap.cc:492] T 10557372aabd490f9b8340c475913aae P d38dc8a8d46744ec8a4c69eadec67f03: Bootstrap starting.
I20260619 18:51:39.073539 21686 tablet_bootstrap.cc:654] T 10557372aabd490f9b8340c475913aae P 0d970abe09234fbb98ad3965c3bc99d7: Neither blocks nor log segments found. Creating new log.
I20260619 18:51:39.073885 21688 tablet_bootstrap.cc:492] T 10557372aabd490f9b8340c475913aae P 4c7959d1a2f54fd3850cd7736a5085f9: Bootstrap starting.
I20260619 18:51:39.074888 21686 log.cc:824] T 10557372aabd490f9b8340c475913aae P 0d970abe09234fbb98ad3965c3bc99d7: Log is configured to *not* fsync() on all Append() calls
I20260619 18:51:39.077332 21688 tablet_bootstrap.cc:654] T 10557372aabd490f9b8340c475913aae P 4c7959d1a2f54fd3850cd7736a5085f9: Neither blocks nor log segments found. Creating new log.
I20260619 18:51:39.077355 21686 tablet_bootstrap.cc:492] T 10557372aabd490f9b8340c475913aae P 0d970abe09234fbb98ad3965c3bc99d7: No bootstrap required, opened a new log
I20260619 18:51:39.077598 21686 ts_tablet_manager.cc:1397] T 10557372aabd490f9b8340c475913aae P 0d970abe09234fbb98ad3965c3bc99d7: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260619 18:51:39.078500 21688 log.cc:824] T 10557372aabd490f9b8340c475913aae P 4c7959d1a2f54fd3850cd7736a5085f9: Log is configured to *not* fsync() on all Append() calls
I20260619 18:51:39.078742 21687 tablet_bootstrap.cc:654] T 10557372aabd490f9b8340c475913aae P d38dc8a8d46744ec8a4c69eadec67f03: Neither blocks nor log segments found. Creating new log.
I20260619 18:51:39.079762 21687 log.cc:824] T 10557372aabd490f9b8340c475913aae P d38dc8a8d46744ec8a4c69eadec67f03: Log is configured to *not* fsync() on all Append() calls
I20260619 18:51:39.081362 21688 tablet_bootstrap.cc:492] T 10557372aabd490f9b8340c475913aae P 4c7959d1a2f54fd3850cd7736a5085f9: No bootstrap required, opened a new log
I20260619 18:51:39.081637 21688 ts_tablet_manager.cc:1397] T 10557372aabd490f9b8340c475913aae P 4c7959d1a2f54fd3850cd7736a5085f9: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260619 18:51:39.082567 21687 tablet_bootstrap.cc:492] T 10557372aabd490f9b8340c475913aae P d38dc8a8d46744ec8a4c69eadec67f03: No bootstrap required, opened a new log
I20260619 18:51:39.082819 21687 ts_tablet_manager.cc:1397] T 10557372aabd490f9b8340c475913aae P d38dc8a8d46744ec8a4c69eadec67f03: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20260619 18:51:39.085738 21686 raft_consensus.cc:348] T 10557372aabd490f9b8340c475913aae P 0d970abe09234fbb98ad3965c3bc99d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38dc8a8d46744ec8a4c69eadec67f03" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 33565 } } peers { permanent_uuid: "4c7959d1a2f54fd3850cd7736a5085f9" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 45187 } } peers { permanent_uuid: "0d970abe09234fbb98ad3965c3bc99d7" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 42687 } }
I20260619 18:51:39.086086 21686 raft_consensus.cc:374] T 10557372aabd490f9b8340c475913aae P 0d970abe09234fbb98ad3965c3bc99d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:51:39.086205 21686 raft_consensus.cc:729] T 10557372aabd490f9b8340c475913aae P 0d970abe09234fbb98ad3965c3bc99d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d970abe09234fbb98ad3965c3bc99d7, State: Initialized, Role: FOLLOWER
I20260619 18:51:39.086756 21686 consensus_queue.cc:260] T 10557372aabd490f9b8340c475913aae P 0d970abe09234fbb98ad3965c3bc99d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38dc8a8d46744ec8a4c69eadec67f03" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 33565 } } peers { permanent_uuid: "4c7959d1a2f54fd3850cd7736a5085f9" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 45187 } } peers { permanent_uuid: "0d970abe09234fbb98ad3965c3bc99d7" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 42687 } }
I20260619 18:51:39.088368 21686 ts_tablet_manager.cc:1428] T 10557372aabd490f9b8340c475913aae P 0d970abe09234fbb98ad3965c3bc99d7: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s
I20260619 18:51:39.091763 21687 raft_consensus.cc:348] T 10557372aabd490f9b8340c475913aae P d38dc8a8d46744ec8a4c69eadec67f03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38dc8a8d46744ec8a4c69eadec67f03" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 33565 } } peers { permanent_uuid: "4c7959d1a2f54fd3850cd7736a5085f9" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 45187 } } peers { permanent_uuid: "0d970abe09234fbb98ad3965c3bc99d7" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 42687 } }
I20260619 18:51:39.092257 21687 raft_consensus.cc:374] T 10557372aabd490f9b8340c475913aae P d38dc8a8d46744ec8a4c69eadec67f03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:51:39.092362 21687 raft_consensus.cc:729] T 10557372aabd490f9b8340c475913aae P d38dc8a8d46744ec8a4c69eadec67f03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d38dc8a8d46744ec8a4c69eadec67f03, State: Initialized, Role: FOLLOWER
I20260619 18:51:39.092270 21688 raft_consensus.cc:348] T 10557372aabd490f9b8340c475913aae P 4c7959d1a2f54fd3850cd7736a5085f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38dc8a8d46744ec8a4c69eadec67f03" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 33565 } } peers { permanent_uuid: "4c7959d1a2f54fd3850cd7736a5085f9" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 45187 } } peers { permanent_uuid: "0d970abe09234fbb98ad3965c3bc99d7" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 42687 } }
I20260619 18:51:39.092681 21688 raft_consensus.cc:374] T 10557372aabd490f9b8340c475913aae P 4c7959d1a2f54fd3850cd7736a5085f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:51:39.092768 21688 raft_consensus.cc:729] T 10557372aabd490f9b8340c475913aae P 4c7959d1a2f54fd3850cd7736a5085f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c7959d1a2f54fd3850cd7736a5085f9, State: Initialized, Role: FOLLOWER
I20260619 18:51:39.092986 21687 consensus_queue.cc:260] T 10557372aabd490f9b8340c475913aae P d38dc8a8d46744ec8a4c69eadec67f03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38dc8a8d46744ec8a4c69eadec67f03" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 33565 } } peers { permanent_uuid: "4c7959d1a2f54fd3850cd7736a5085f9" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 45187 } } peers { permanent_uuid: "0d970abe09234fbb98ad3965c3bc99d7" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 42687 } }
I20260619 18:51:39.093341 21688 consensus_queue.cc:260] T 10557372aabd490f9b8340c475913aae P 4c7959d1a2f54fd3850cd7736a5085f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38dc8a8d46744ec8a4c69eadec67f03" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 33565 } } peers { permanent_uuid: "4c7959d1a2f54fd3850cd7736a5085f9" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 45187 } } peers { permanent_uuid: "0d970abe09234fbb98ad3965c3bc99d7" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 42687 } }
I20260619 18:51:39.095665 21535 heartbeater.cc:503] Master 127.3.243.126:42681 was elected leader, sending a full tablet report...
I20260619 18:51:39.096225 21668 heartbeater.cc:503] Master 127.3.243.126:42681 was elected leader, sending a full tablet report...
I20260619 18:51:39.096315 21688 ts_tablet_manager.cc:1428] T 10557372aabd490f9b8340c475913aae P 4c7959d1a2f54fd3850cd7736a5085f9: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20260619 18:51:39.096948 21687 ts_tablet_manager.cc:1428] T 10557372aabd490f9b8340c475913aae P d38dc8a8d46744ec8a4c69eadec67f03: Time spent starting tablet: real 0.014s user 0.011s sys 0.004s
I20260619 18:51:39.136106 21693 raft_consensus.cc:482] T 10557372aabd490f9b8340c475913aae P d38dc8a8d46744ec8a4c69eadec67f03 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:51:39.136420 21693 raft_consensus.cc:504] T 10557372aabd490f9b8340c475913aae P d38dc8a8d46744ec8a4c69eadec67f03 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38dc8a8d46744ec8a4c69eadec67f03" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 33565 } } peers { permanent_uuid: "4c7959d1a2f54fd3850cd7736a5085f9" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 45187 } } peers { permanent_uuid: "0d970abe09234fbb98ad3965c3bc99d7" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 42687 } }
I20260619 18:51:39.138127 21693 leader_election.cc:288] T 10557372aabd490f9b8340c475913aae P d38dc8a8d46744ec8a4c69eadec67f03 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c7959d1a2f54fd3850cd7736a5085f9 (127.3.243.66:45187), 0d970abe09234fbb98ad3965c3bc99d7 (127.3.243.65:42687)
I20260619 18:51:39.144418 21685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:39.138542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.67:56187 (local address 127.3.243.66:45187)
0619 18:51:39.138757 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0619 18:51:39.138763 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:51:39.138987 (+ 224us) server_negotiation.cc:407] Connection header received
0619 18:51:39.139104 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:51:39.139110 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:51:39.139188 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:51:39.139289 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:51:39.139629 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:39.140579 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:39.142553 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:39.143530 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:39.143668 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:39.143740 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 18:51:39.144013 (+ 273us) server_negotiation.cc:299] Negotiation successful
0619 18:51:39.144136 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260619 18:51:39.144419 21670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:39.138411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.66:45187 (local address 127.3.243.67:56187)
0619 18:51:39.138773 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:39.138795 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 18:51:39.138970 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:51:39.139328 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:51:39.139332 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:51:39.139356 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:51:39.139515 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:51:39.139522 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:51:39.140696 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:51:39.140702 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:51:39.142381 (+ 1679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:51:39.142395 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:51:39.143668 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:51:39.143674 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:51:39.143859 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:39.143882 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 18:51:39.143967 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 18:51:39.144095 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"mutex_wait_us":34}
I20260619 18:51:39.144941 21683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:39.138732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.67:56133 (local address 127.3.243.65:42687)
0619 18:51:39.138931 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0619 18:51:39.138937 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:51:39.139510 (+ 573us) server_negotiation.cc:407] Connection header received
0619 18:51:39.139686 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:51:39.139690 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:51:39.139746 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:51:39.139810 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:51:39.140360 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:39.141312 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:39.143238 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:39.144063 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:39.144232 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:39.144302 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 18:51:39.144605 (+ 303us) server_negotiation.cc:299] Negotiation successful
0619 18:51:39.144726 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 18:51:39.145469 21695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:39.138487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.65:42687 (local address 127.3.243.67:56133)
0619 18:51:39.139407 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:39.139423 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 18:51:39.139545 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:51:39.139846 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:51:39.139850 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:51:39.139874 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:51:39.140225 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:51:39.140237 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:51:39.141461 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:51:39.141465 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:51:39.143070 (+ 1605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:51:39.143081 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:51:39.144267 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:51:39.144273 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:51:39.144431 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:39.144453 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 18:51:39.145102 (+ 649us) client_negotiation.cc:240] Negotiation successful
0619 18:51:39.145216 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":789,"mutex_wait_us":19,"thread_start_us":580,"threads_started":1}
I20260619 18:51:39.145362 21490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10557372aabd490f9b8340c475913aae" candidate_uuid: "d38dc8a8d46744ec8a4c69eadec67f03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c7959d1a2f54fd3850cd7736a5085f9" is_pre_election: true
I20260619 18:51:39.145838 21490 raft_consensus.cc:2454] T 10557372aabd490f9b8340c475913aae P 4c7959d1a2f54fd3850cd7736a5085f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d38dc8a8d46744ec8a4c69eadec67f03 in term 0.
I20260619 18:51:39.146333 21356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10557372aabd490f9b8340c475913aae" candidate_uuid: "d38dc8a8d46744ec8a4c69eadec67f03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d970abe09234fbb98ad3965c3bc99d7" is_pre_election: true
I20260619 18:51:39.146459 21556 leader_election.cc:302] T 10557372aabd490f9b8340c475913aae P d38dc8a8d46744ec8a4c69eadec67f03 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c7959d1a2f54fd3850cd7736a5085f9, d38dc8a8d46744ec8a4c69eadec67f03; no voters:
I20260619 18:51:39.146804 21693 raft_consensus.cc:2781] T 10557372aabd490f9b8340c475913aae P d38dc8a8d46744ec8a4c69eadec67f03 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:51:39.146821 21356 raft_consensus.cc:2454] T 10557372aabd490f9b8340c475913aae P 0d970abe09234fbb98ad3965c3bc99d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d38dc8a8d46744ec8a4c69eadec67f03 in term 0.
I20260619 18:51:39.146963 21693 raft_consensus.cc:482] T 10557372aabd490f9b8340c475913aae P d38dc8a8d46744ec8a4c69eadec67f03 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:51:39.147071 21693 raft_consensus.cc:3037] T 10557372aabd490f9b8340c475913aae P d38dc8a8d46744ec8a4c69eadec67f03 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:51:39.149264 21693 raft_consensus.cc:504] T 10557372aabd490f9b8340c475913aae P d38dc8a8d46744ec8a4c69eadec67f03 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38dc8a8d46744ec8a4c69eadec67f03" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 33565 } } peers { permanent_uuid: "4c7959d1a2f54fd3850cd7736a5085f9" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 45187 } } peers { permanent_uuid: "0d970abe09234fbb98ad3965c3bc99d7" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 42687 } }
I20260619 18:51:39.149786 21693 leader_election.cc:288] T 10557372aabd490f9b8340c475913aae P d38dc8a8d46744ec8a4c69eadec67f03 [CANDIDATE]: Term 1 election: Requested vote from peers 4c7959d1a2f54fd3850cd7736a5085f9 (127.3.243.66:45187), 0d970abe09234fbb98ad3965c3bc99d7 (127.3.243.65:42687)
I20260619 18:51:39.150197 21356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10557372aabd490f9b8340c475913aae" candidate_uuid: "d38dc8a8d46744ec8a4c69eadec67f03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d970abe09234fbb98ad3965c3bc99d7"
I20260619 18:51:39.150194 21490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10557372aabd490f9b8340c475913aae" candidate_uuid: "d38dc8a8d46744ec8a4c69eadec67f03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c7959d1a2f54fd3850cd7736a5085f9"
I20260619 18:51:39.150357 21356 raft_consensus.cc:3037] T 10557372aabd490f9b8340c475913aae P 0d970abe09234fbb98ad3965c3bc99d7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:51:39.150370 21490 raft_consensus.cc:3037] T 10557372aabd490f9b8340c475913aae P 4c7959d1a2f54fd3850cd7736a5085f9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:51:39.152462 21356 raft_consensus.cc:2454] T 10557372aabd490f9b8340c475913aae P 0d970abe09234fbb98ad3965c3bc99d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d38dc8a8d46744ec8a4c69eadec67f03 in term 1.
I20260619 18:51:39.152557 21490 raft_consensus.cc:2454] T 10557372aabd490f9b8340c475913aae P 4c7959d1a2f54fd3850cd7736a5085f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d38dc8a8d46744ec8a4c69eadec67f03 in term 1.
I20260619 18:51:39.152848 21559 leader_election.cc:302] T 10557372aabd490f9b8340c475913aae P d38dc8a8d46744ec8a4c69eadec67f03 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d970abe09234fbb98ad3965c3bc99d7, d38dc8a8d46744ec8a4c69eadec67f03; no voters:
I20260619 18:51:39.153172 21693 raft_consensus.cc:2781] T 10557372aabd490f9b8340c475913aae P d38dc8a8d46744ec8a4c69eadec67f03 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:51:39.153584 21693 raft_consensus.cc:686] T 10557372aabd490f9b8340c475913aae P d38dc8a8d46744ec8a4c69eadec67f03 [term 1 LEADER]: Becoming Leader. State: Replica: d38dc8a8d46744ec8a4c69eadec67f03, State: Running, Role: LEADER
I20260619 18:51:39.154026 21693 consensus_queue.cc:237] T 10557372aabd490f9b8340c475913aae P d38dc8a8d46744ec8a4c69eadec67f03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38dc8a8d46744ec8a4c69eadec67f03" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 33565 } } peers { permanent_uuid: "4c7959d1a2f54fd3850cd7736a5085f9" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 45187 } } peers { permanent_uuid: "0d970abe09234fbb98ad3965c3bc99d7" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 42687 } }
I20260619 18:51:39.158630 21212 catalog_manager.cc:5495] T 10557372aabd490f9b8340c475913aae P d38dc8a8d46744ec8a4c69eadec67f03 reported cstate change: term changed from 0 to 1, leader changed from <none> to d38dc8a8d46744ec8a4c69eadec67f03 (127.3.243.67). New cstate: current_term: 1 leader_uuid: "d38dc8a8d46744ec8a4c69eadec67f03" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38dc8a8d46744ec8a4c69eadec67f03" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 33565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c7959d1a2f54fd3850cd7736a5085f9" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 45187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d970abe09234fbb98ad3965c3bc99d7" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 42687 } health_report { overall_health: UNKNOWN } } }
I20260619 18:51:39.177853 21681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:39.173047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36468 (local address 127.3.243.67:33565)
0619 18:51:39.173237 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 18:51:39.173244 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:51:39.173299 (+ 55us) server_negotiation.cc:407] Connection header received
0619 18:51:39.173470 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:51:39.173474 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:51:39.173547 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:51:39.173663 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:51:39.173936 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:39.174647 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:39.175983 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:39.176425 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:39.176515 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:39.176923 (+ 408us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:51:39.177391 (+ 468us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:51:39.177517 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 18:51:39.177552 (+ 35us) server_negotiation.cc:299] Negotiation successful
0619 18:51:39.177612 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 18:51:39.185530 21490 raft_consensus.cc:1261] T 10557372aabd490f9b8340c475913aae P 4c7959d1a2f54fd3850cd7736a5085f9 [term 1 FOLLOWER]: Refusing update from remote peer d38dc8a8d46744ec8a4c69eadec67f03: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:51:39.185528 21356 raft_consensus.cc:1261] T 10557372aabd490f9b8340c475913aae P 0d970abe09234fbb98ad3965c3bc99d7 [term 1 FOLLOWER]: Refusing update from remote peer d38dc8a8d46744ec8a4c69eadec67f03: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:51:39.186475 21696 consensus_queue.cc:1035] T 10557372aabd490f9b8340c475913aae P d38dc8a8d46744ec8a4c69eadec67f03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c7959d1a2f54fd3850cd7736a5085f9" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 45187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:51:39.186826 21693 consensus_queue.cc:1035] T 10557372aabd490f9b8340c475913aae P d38dc8a8d46744ec8a4c69eadec67f03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d970abe09234fbb98ad3965c3bc99d7" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 42687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:51:39.199939 21700 mvcc.cc:205] Tried to move back new op lower bound from 7298642326254088192 to 7298642326138417152. Current Snapshot: MvccSnapshot[applied={T|T < 7298642326254088192}]
I20260619 18:51:39.209769 21212 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50432:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:51:39.209997 21212 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50432:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:51:39.213316 21212 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 24f8c13db7404fe0a7efd4631402409a: Sending DeleteTablet for 3 replicas of tablet 10557372aabd490f9b8340c475913aae
I20260619 18:51:39.214407 21603 tablet_service.cc:1514] Processing DeleteTablet for tablet 10557372aabd490f9b8340c475913aae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:51:39 UTC) from {username='kudu', principal='kudu/127.3.243.126@KRBTEST.COM'} at 127.0.0.1:36458
I20260619 18:51:39.214464 21336 tablet_service.cc:1514] Processing DeleteTablet for tablet 10557372aabd490f9b8340c475913aae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:51:39 UTC) from {username='kudu', principal='kudu/127.3.243.126@KRBTEST.COM'} at 127.0.0.1:39210
I20260619 18:51:39.214648 21470 tablet_service.cc:1514] Processing DeleteTablet for tablet 10557372aabd490f9b8340c475913aae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:51:39 UTC) from {username='kudu', principal='kudu/127.3.243.126@KRBTEST.COM'} at 127.0.0.1:54912
I20260619 18:51:39.217001 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 21273
I20260619 18:51:39.225775 21710 tablet_replica.cc:330] T 10557372aabd490f9b8340c475913aae P 4c7959d1a2f54fd3850cd7736a5085f9: stopping tablet replica
I20260619 18:51:39.225986 21708 tablet_replica.cc:330] T 10557372aabd490f9b8340c475913aae P d38dc8a8d46744ec8a4c69eadec67f03: stopping tablet replica
I20260619 18:51:39.226326 21710 raft_consensus.cc:2229] T 10557372aabd490f9b8340c475913aae P 4c7959d1a2f54fd3850cd7736a5085f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:51:39.226399 21708 raft_consensus.cc:2229] T 10557372aabd490f9b8340c475913aae P d38dc8a8d46744ec8a4c69eadec67f03 [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:51:39.226922 21710 raft_consensus.cc:2258] T 10557372aabd490f9b8340c475913aae P 4c7959d1a2f54fd3850cd7736a5085f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:51:39.227138 21708 raft_consensus.cc:2258] T 10557372aabd490f9b8340c475913aae P d38dc8a8d46744ec8a4c69eadec67f03 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:51:39.227612 21709 tablet_replica.cc:330] T 10557372aabd490f9b8340c475913aae P 0d970abe09234fbb98ad3965c3bc99d7: stopping tablet replica
I20260619 18:51:39.228057 21709 raft_consensus.cc:2229] T 10557372aabd490f9b8340c475913aae P 0d970abe09234fbb98ad3965c3bc99d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:51:39.228578 21709 raft_consensus.cc:2258] T 10557372aabd490f9b8340c475913aae P 0d970abe09234fbb98ad3965c3bc99d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:51:39.229290 21708 ts_tablet_manager.cc:1905] T 10557372aabd490f9b8340c475913aae P d38dc8a8d46744ec8a4c69eadec67f03: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:51:39.230378 21709 ts_tablet_manager.cc:1905] T 10557372aabd490f9b8340c475913aae P 0d970abe09234fbb98ad3965c3bc99d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:51:39.231750 21710 ts_tablet_manager.cc:1905] T 10557372aabd490f9b8340c475913aae P 4c7959d1a2f54fd3850cd7736a5085f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:51:39.236876 21709 ts_tablet_manager.cc:1918] T 10557372aabd490f9b8340c475913aae P 0d970abe09234fbb98ad3965c3bc99d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:51:39.237018 21708 ts_tablet_manager.cc:1918] T 10557372aabd490f9b8340c475913aae P d38dc8a8d46744ec8a4c69eadec67f03: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:51:39.237107 21709 log.cc:1196] T 10557372aabd490f9b8340c475913aae P 0d970abe09234fbb98ad3965c3bc99d7: Deleting WAL directory at /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894810715282-4045-0/minicluster-data/ts-0/wal/wals/10557372aabd490f9b8340c475913aae
I20260619 18:51:39.237210 21708 log.cc:1196] T 10557372aabd490f9b8340c475913aae P d38dc8a8d46744ec8a4c69eadec67f03: Deleting WAL directory at /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894810715282-4045-0/minicluster-data/ts-2/wal/wals/10557372aabd490f9b8340c475913aae
I20260619 18:51:39.237664 21709 ts_tablet_manager.cc:1939] T 10557372aabd490f9b8340c475913aae P 0d970abe09234fbb98ad3965c3bc99d7: Deleting consensus metadata
I20260619 18:51:39.237816 21708 ts_tablet_manager.cc:1939] T 10557372aabd490f9b8340c475913aae P d38dc8a8d46744ec8a4c69eadec67f03: Deleting consensus metadata
I20260619 18:51:39.239493 21196 catalog_manager.cc:4841] TS d38dc8a8d46744ec8a4c69eadec67f03 (127.3.243.67:33565): tablet 10557372aabd490f9b8340c475913aae (table test-table [id=45f1a2230ea94bff865c1fb10d2a5c9f]) successfully deleted
I20260619 18:51:39.239452 21199 catalog_manager.cc:4841] TS 0d970abe09234fbb98ad3965c3bc99d7 (127.3.243.65:42687): tablet 10557372aabd490f9b8340c475913aae (table test-table [id=45f1a2230ea94bff865c1fb10d2a5c9f]) successfully deleted
I20260619 18:51:39.242939 21710 ts_tablet_manager.cc:1918] T 10557372aabd490f9b8340c475913aae P 4c7959d1a2f54fd3850cd7736a5085f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:51:39.243083 21710 log.cc:1196] T 10557372aabd490f9b8340c475913aae P 4c7959d1a2f54fd3850cd7736a5085f9: Deleting WAL directory at /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894810715282-4045-0/minicluster-data/ts-1/wal/wals/10557372aabd490f9b8340c475913aae
I20260619 18:51:39.243508 21710 ts_tablet_manager.cc:1939] T 10557372aabd490f9b8340c475913aae P 4c7959d1a2f54fd3850cd7736a5085f9: Deleting consensus metadata
I20260619 18:51:39.245226 21196 catalog_manager.cc:4841] TS 4c7959d1a2f54fd3850cd7736a5085f9 (127.3.243.66:45187): tablet 10557372aabd490f9b8340c475913aae (table test-table [id=45f1a2230ea94bff865c1fb10d2a5c9f]) successfully deleted
Jun 19 18:51:39 dist-test-slave-m9wg krb5kdc[21167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895096, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.65@KRBTEST.COM
I20260619 18:51:39.254438 21683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:39.218330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39218 (local address 127.3.243.65:42687)
0619 18:51:39.218505 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 18:51:39.218511 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:51:39.237516 (+ 19005us) server_negotiation.cc:407] Connection header received
0619 18:51:39.237984 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:51:39.237991 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:51:39.238073 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:51:39.238287 (+ 214us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:51:39.240571 (+ 2284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:39.241614 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:39.243467 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:39.244058 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:39.244235 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:39.249140 (+ 4905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:51:39.249152 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:51:39.249165 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:51:39.249200 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:51:39.251781 (+ 2581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:51:39.252621 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:51:39.252628 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:51:39.252630 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:51:39.252746 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:51:39.253095 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:51:39.253098 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:51:39.253100 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:51:39.253431 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:51:39.253643 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0619 18:51:39.253971 (+ 328us) server_negotiation.cc:299] Negotiation successful
0619 18:51:39.254146 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 18:51:39.255133 21396 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:51:39.342299 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 21273
I20260619 18:51:39.356712 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 21407
Jun 19 18:51:39 dist-test-slave-m9wg krb5kdc[21167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895096, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.66@KRBTEST.COM
I20260619 18:51:39.371387 21685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:39.357511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54920 (local address 127.3.243.66:45187)
0619 18:51:39.357680 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 18:51:39.357686 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:51:39.357771 (+ 85us) server_negotiation.cc:407] Connection header received
0619 18:51:39.357981 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:51:39.357986 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:51:39.358059 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:51:39.358142 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:51:39.359413 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:39.360138 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:39.361413 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:39.361802 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:39.361918 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:39.366150 (+ 4232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:51:39.366165 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:51:39.366180 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:51:39.366214 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:51:39.368902 (+ 2688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:51:39.369643 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:51:39.369650 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:51:39.369652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:51:39.369758 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:51:39.370099 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:51:39.370102 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:51:39.370104 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:51:39.370439 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:51:39.370653 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0619 18:51:39.370955 (+ 302us) server_negotiation.cc:299] Negotiation successful
0619 18:51:39.371114 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 18:51:39.372038 21530 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:51:39.517171 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 21407
I20260619 18:51:39.532845 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 21540
Jun 19 18:51:39 dist-test-slave-m9wg krb5kdc[21167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895096, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.67@KRBTEST.COM
I20260619 18:51:39.549580 21681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:39.533659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36470 (local address 127.3.243.67:33565)
0619 18:51:39.533861 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0619 18:51:39.533868 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:51:39.533890 (+ 22us) server_negotiation.cc:407] Connection header received
0619 18:51:39.534086 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:51:39.534091 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:51:39.534209 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:51:39.534332 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:51:39.535539 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:39.536752 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:39.537921 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:39.538587 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:39.538703 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:39.543054 (+ 4351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:51:39.543074 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:51:39.543094 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:51:39.543144 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:51:39.546829 (+ 3685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:51:39.547455 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:51:39.547464 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:51:39.547468 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:51:39.547671 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:51:39.547993 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:51:39.547998 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:51:39.548002 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:51:39.548460 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:51:39.548801 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0619 18:51:39.548952 (+ 151us) server_negotiation.cc:299] Negotiation successful
0619 18:51:39.549168 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 18:51:39.550420 21663 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:51:39.671079 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 21540
I20260619 18:51:39.686662 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 21180
I20260619 18:51:39.687796 21242 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:51:39.793397 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 21180
2026-06-19T18:51:39Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (3959 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-19T18:51:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:51:39Z Disabled control of system clock
I20260619 18:51:39.863636 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894810715282-4045-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894810715282-4045-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894810715282-4045-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.126:33449
--webserver_interface=127.3.243.126
--webserver_port=0
--builtin_ntp_servers=127.3.243.84:43173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.126:33449
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 18:51:40.175168 21719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:51:40.175431 21719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:51:40.175552 21719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:51:40.182564 21719 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:51:40.182658 21719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:51:40.182722 21719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:51:40.182796 21719 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:51:40.182852 21719 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:51:40.191583 21719 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:43173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894810715282-4045-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894810715282-4045-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.243.126:33449
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.126:33449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.243.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894810715282-4045-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:51:40.192806 21719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:51:40.194551 21719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:51:40.205950 21724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:51:40.205951 21725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:51:40.206700 21719 server_base.cc:1034] running on GCE node
W20260619 18:51:40.207391 21727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:51:40.208271 21719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:51:40.209579 21719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:51:40.210764 21719 hybrid_clock.cc:648] HybridClock initialized: now 1781895100210726 us; error 33 us; skew 500 ppm
I20260619 18:51:40.211086 21719 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 18:51:40.213155 21719 webserver.cc:460] Webserver started at http://127.3.243.126:41277/ using document root <none> and password file <none>
I20260619 18:51:40.213778 21719 fs_manager.cc:359] Metadata directory not provided
I20260619 18:51:40.213865 21719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:51:40.214154 21719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:51:40.216917 21719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894810715282-4045-0/minicluster-data/master-0/data/instance:
uuid: "7e97bcf7ebbd4ba7a1d02c305fb5d186"
format_stamp: "Formatted at 2026-06-19 18:51:40 on dist-test-slave-m9wg"
server_key: "52a80e14afcc043943aab92a251c9dd8"
server_key_iv: "6223fd211047f3d12f2574662d0fd76b"
server_key_version: "encryptionkey@0"
I20260619 18:51:40.217664 21719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894810715282-4045-0/minicluster-data/master-0/wal/instance:
uuid: "7e97bcf7ebbd4ba7a1d02c305fb5d186"
format_stamp: "Formatted at 2026-06-19 18:51:40 on dist-test-slave-m9wg"
server_key: "52a80e14afcc043943aab92a251c9dd8"
server_key_iv: "6223fd211047f3d12f2574662d0fd76b"
server_key_version: "encryptionkey@0"
I20260619 18:51:40.222303 21719 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 18:51:40.225394 21733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:51:40.226531 21719 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 18:51:40.226691 21719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894810715282-4045-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894810715282-4045-0/minicluster-data/master-0/wal
uuid: "7e97bcf7ebbd4ba7a1d02c305fb5d186"
format_stamp: "Formatted at 2026-06-19 18:51:40 on dist-test-slave-m9wg"
server_key: "52a80e14afcc043943aab92a251c9dd8"
server_key_iv: "6223fd211047f3d12f2574662d0fd76b"
server_key_version: "encryptionkey@0"
I20260619 18:51:40.226871 21719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894810715282-4045-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894810715282-4045-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894810715282-4045-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:51:40.249897 21719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:51:40.250653 21719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:51:40.250912 21719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:51:40.267575 21719 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.126:33449
I20260619 18:51:40.267594 21784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.126:33449 every 8 connection(s)
I20260619 18:51:40.268836 21719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
I20260619 18:51:40.272346 21785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:51:40.272665 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 21719
I20260619 18:51:40.272845 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894810715282-4045-0/minicluster-data/master-0/wal/instance
I20260619 18:51:40.273388 4045 external_mini_cluster.cc:1428] Setting key 7882243e85e62e13698093000f36b7f2
I20260619 18:51:40.285287 21785 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e97bcf7ebbd4ba7a1d02c305fb5d186: Bootstrap starting.
I20260619 18:51:40.286758 21788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:40.275221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47648 (local address 127.3.243.126:33449)
0619 18:51:40.276240 (+ 1019us) server_negotiation.cc:206] Beginning negotiation
0619 18:51:40.276250 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:51:40.276294 (+ 44us) server_negotiation.cc:407] Connection header received
0619 18:51:40.278309 (+ 2015us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:51:40.278330 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:51:40.278767 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:51:40.279076 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:51:40.279608 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:40.280893 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:40.282774 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:40.283417 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:40.283539 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:40.284699 (+ 1160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:51:40.284706 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:51:40.284719 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:51:40.285211 (+ 492us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:51:40.285288 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:51:40.285309 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:51:40.285401 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 18:51:40.285599 (+ 198us) server_negotiation.cc:299] Negotiation successful
0619 18:51:40.285882 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":381,"threads_started":1}
I20260619 18:51:40.288415 21785 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e97bcf7ebbd4ba7a1d02c305fb5d186: Neither blocks nor log segments found. Creating new log.
I20260619 18:51:40.289372 21785 log.cc:824] T 00000000000000000000000000000000 P 7e97bcf7ebbd4ba7a1d02c305fb5d186: Log is configured to *not* fsync() on all Append() calls
I20260619 18:51:40.291725 21785 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e97bcf7ebbd4ba7a1d02c305fb5d186: No bootstrap required, opened a new log
I20260619 18:51:40.298514 21785 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e97bcf7ebbd4ba7a1d02c305fb5d186 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e97bcf7ebbd4ba7a1d02c305fb5d186" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 33449 } }
I20260619 18:51:40.298840 21785 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7e97bcf7ebbd4ba7a1d02c305fb5d186 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:51:40.298931 21785 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e97bcf7ebbd4ba7a1d02c305fb5d186 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e97bcf7ebbd4ba7a1d02c305fb5d186, State: Initialized, Role: FOLLOWER
I20260619 18:51:40.299489 21785 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e97bcf7ebbd4ba7a1d02c305fb5d186 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e97bcf7ebbd4ba7a1d02c305fb5d186" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 33449 } }
I20260619 18:51:40.299700 21785 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7e97bcf7ebbd4ba7a1d02c305fb5d186 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:51:40.299832 21785 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e97bcf7ebbd4ba7a1d02c305fb5d186 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:51:40.299980 21785 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e97bcf7ebbd4ba7a1d02c305fb5d186 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:51:40.302151 21785 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e97bcf7ebbd4ba7a1d02c305fb5d186 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e97bcf7ebbd4ba7a1d02c305fb5d186" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 33449 } }
I20260619 18:51:40.302610 21785 leader_election.cc:302] T 00000000000000000000000000000000 P 7e97bcf7ebbd4ba7a1d02c305fb5d186 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e97bcf7ebbd4ba7a1d02c305fb5d186; no voters:
I20260619 18:51:40.303081 21785 leader_election.cc:288] T 00000000000000000000000000000000 P 7e97bcf7ebbd4ba7a1d02c305fb5d186 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:51:40.303211 21790 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e97bcf7ebbd4ba7a1d02c305fb5d186 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:51:40.303856 21790 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7e97bcf7ebbd4ba7a1d02c305fb5d186 [term 1 LEADER]: Becoming Leader. State: Replica: 7e97bcf7ebbd4ba7a1d02c305fb5d186, State: Running, Role: LEADER
I20260619 18:51:40.304414 21790 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e97bcf7ebbd4ba7a1d02c305fb5d186 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e97bcf7ebbd4ba7a1d02c305fb5d186" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 33449 } }
I20260619 18:51:40.304976 21785 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e97bcf7ebbd4ba7a1d02c305fb5d186 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:51:40.306893 21792 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e97bcf7ebbd4ba7a1d02c305fb5d186 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e97bcf7ebbd4ba7a1d02c305fb5d186. Latest consensus state: current_term: 1 leader_uuid: "7e97bcf7ebbd4ba7a1d02c305fb5d186" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e97bcf7ebbd4ba7a1d02c305fb5d186" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 33449 } } }
I20260619 18:51:40.307317 21792 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e97bcf7ebbd4ba7a1d02c305fb5d186 [sys.catalog]: This master's current role is: LEADER
I20260619 18:51:40.308286 21791 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e97bcf7ebbd4ba7a1d02c305fb5d186 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7e97bcf7ebbd4ba7a1d02c305fb5d186" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e97bcf7ebbd4ba7a1d02c305fb5d186" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 33449 } } }
I20260619 18:51:40.308670 21791 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e97bcf7ebbd4ba7a1d02c305fb5d186 [sys.catalog]: This master's current role is: LEADER
I20260619 18:51:40.310935 21799 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:51:40.315361 21799 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:51:40.323071 21799 catalog_manager.cc:1347] Generated new cluster ID: cf95f484fab44804862b19a525b31df7
I20260619 18:51:40.323182 21799 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:51:40.339025 21799 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:51:40.340795 21799 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:51:40.353706 21799 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7e97bcf7ebbd4ba7a1d02c305fb5d186: Generated new TSK 0
I20260619 18:51:40.354760 21799 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:51:40.372442 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894810715282-4045-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894810715282-4045-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894810715282-4045-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894810715282-4045-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.65:0
--local_ip_for_outbound_sockets=127.3.243.65
--webserver_interface=127.3.243.65
--webserver_port=0
--tserver_master_addrs=127.3.243.126:33449
--builtin_ntp_servers=127.3.243.84:43173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 18:51:40.683535 21809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:51:40.683780 21809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:51:40.683897 21809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:51:40.690784 21809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:51:40.690866 21809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:51:40.690997 21809 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.65
I20260619 18:51:40.699621 21809 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:43173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894810715282-4045-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894810715282-4045-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894810715282-4045-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.243.65
--webserver_port=0
--tserver_master_addrs=127.3.243.126:33449
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.65
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894810715282-4045-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:51:40.700845 21809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:51:40.702533 21809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:51:40.714501 21814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:51:40.714570 21817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:51:40.714603 21815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:51:40.715276 21809 server_base.cc:1034] running on GCE node
I20260619 18:51:40.715821 21809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:51:40.716761 21809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:51:40.718012 21809 hybrid_clock.cc:648] HybridClock initialized: now 1781895100717953 us; error 41 us; skew 500 ppm
I20260619 18:51:40.718325 21809 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 18:51:40.720481 21809 webserver.cc:460] Webserver started at http://127.3.243.65:42605/ using document root <none> and password file <none>
I20260619 18:51:40.721122 21809 fs_manager.cc:359] Metadata directory not provided
I20260619 18:51:40.721206 21809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:51:40.721460 21809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:51:40.724076 21809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894810715282-4045-0/minicluster-data/ts-0/data/instance:
uuid: "61ad01072f68435aab5d63e0bf38e9cc"
format_stamp: "Formatted at 2026-06-19 18:51:40 on dist-test-slave-m9wg"
server_key: "3fcbabfb07975407769bdcbde53ff6c9"
server_key_iv: "02bdeb9202779292b458e3f578a6501c"
server_key_version: "encryptionkey@0"
I20260619 18:51:40.724841 21809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894810715282-4045-0/minicluster-data/ts-0/wal/instance:
uuid: "61ad01072f68435aab5d63e0bf38e9cc"
format_stamp: "Formatted at 2026-06-19 18:51:40 on dist-test-slave-m9wg"
server_key: "3fcbabfb07975407769bdcbde53ff6c9"
server_key_iv: "02bdeb9202779292b458e3f578a6501c"
server_key_version: "encryptionkey@0"
I20260619 18:51:40.729341 21809 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 18:51:40.732532 21823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:51:40.733760 21809 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s
I20260619 18:51:40.733896 21809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894810715282-4045-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894810715282-4045-0/minicluster-data/ts-0/wal
uuid: "61ad01072f68435aab5d63e0bf38e9cc"
format_stamp: "Formatted at 2026-06-19 18:51:40 on dist-test-slave-m9wg"
server_key: "3fcbabfb07975407769bdcbde53ff6c9"
server_key_iv: "02bdeb9202779292b458e3f578a6501c"
server_key_version: "encryptionkey@0"
I20260619 18:51:40.734087 21809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894810715282-4045-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894810715282-4045-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894810715282-4045-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:51:40.755928 21809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:51:40.756623 21809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:51:40.756860 21809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:51:40.757666 21809 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:51:40.758962 21809 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:51:40.759042 21809 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:51:40.759116 21809 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:51:40.759167 21809 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:51:40.787534 21809 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.65:33439
I20260619 18:51:40.787608 21935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.65:33439 every 8 connection(s)
I20260619 18:51:40.788810 21809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894810715282-4045-0/minicluster-data/ts-0/data/info.pb
I20260619 18:51:40.789516 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 21809
I20260619 18:51:40.789678 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894810715282-4045-0/minicluster-data/ts-0/wal/instance
I20260619 18:51:40.790213 4045 external_mini_cluster.cc:1428] Setting key 15e181d12dbd7e2d5cb1f697cf15dce3
I20260619 18:51:40.793195 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894810715282-4045-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894810715282-4045-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894810715282-4045-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894810715282-4045-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.66:0
--local_ip_for_outbound_sockets=127.3.243.66
--webserver_interface=127.3.243.66
--webserver_port=0
--tserver_master_addrs=127.3.243.126:33449
--builtin_ntp_servers=127.3.243.84:43173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 18:51:40.802361 21938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:40.791521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.65:33655 (local address 127.3.243.126:33449)
0619 18:51:40.791986 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0619 18:51:40.791992 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:51:40.793427 (+ 1435us) server_negotiation.cc:407] Connection header received
0619 18:51:40.795668 (+ 2241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:51:40.795672 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:51:40.795736 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:51:40.795825 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:51:40.797026 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:40.797730 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:40.799733 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:40.800125 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:40.800265 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:40.801568 (+ 1303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:51:40.801572 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:51:40.801575 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:51:40.801593 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:51:40.801629 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:51:40.801641 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:51:40.801713 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 18:51:40.801942 (+ 229us) server_negotiation.cc:299] Negotiation successful
0619 18:51:40.802037 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":217,"threads_started":1}
I20260619 18:51:40.803313 21939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:40.792128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:33449 (local address 127.3.243.65:33655)
0619 18:51:40.793212 (+ 1084us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:40.793257 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 18:51:40.795402 (+ 2145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:51:40.796058 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:51:40.796067 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:51:40.796209 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:51:40.796886 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:51:40.796907 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:51:40.797883 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:51:40.797888 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:51:40.799614 (+ 1726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:51:40.799627 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:51:40.800329 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:51:40.800335 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:51:40.800596 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:40.801269 (+ 673us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:51:40.801271 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:51:40.801309 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:51:40.801389 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:51:40.801728 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:51:40.801739 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:51:40.801747 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 18:51:40.801903 (+ 156us) client_negotiation.cc:240] Negotiation successful
0619 18:51:40.802274 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":314,"threads_started":1}
I20260619 18:51:40.804837 21936 heartbeater.cc:347] Connected to a master server at 127.3.243.126:33449
I20260619 18:51:40.805104 21936 heartbeater.cc:464] Registering TS with master...
I20260619 18:51:40.805785 21936 heartbeater.cc:511] Master 127.3.243.126:33449 requested a full tablet report, sending...
I20260619 18:51:40.807698 21750 ts_manager.cc:194] Registered new tserver with Master: 61ad01072f68435aab5d63e0bf38e9cc (127.3.243.65:33439)
I20260619 18:51:40.809226 21750 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.243.65:33655
W20260619 18:51:41.097707 21940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:51:41.097970 21940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:51:41.098097 21940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:51:41.105029 21940 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:51:41.105119 21940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:51:41.105259 21940 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.66
I20260619 18:51:41.113857 21940 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:43173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894810715282-4045-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894810715282-4045-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894810715282-4045-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.243.66
--webserver_port=0
--tserver_master_addrs=127.3.243.126:33449
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.66
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894810715282-4045-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:51:41.114986 21940 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:51:41.116815 21940 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:51:41.128510 21946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:51:41.128593 21945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:51:41.128613 21948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:51:41.129218 21940 server_base.cc:1034] running on GCE node
I20260619 18:51:41.129702 21940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:51:41.130559 21940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:51:41.131750 21940 hybrid_clock.cc:648] HybridClock initialized: now 1781895101131710 us; error 49 us; skew 500 ppm
I20260619 18:51:41.132022 21940 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 18:51:41.133960 21940 webserver.cc:460] Webserver started at http://127.3.243.66:44827/ using document root <none> and password file <none>
I20260619 18:51:41.134552 21940 fs_manager.cc:359] Metadata directory not provided
I20260619 18:51:41.134642 21940 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:51:41.134928 21940 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:51:41.137496 21940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894810715282-4045-0/minicluster-data/ts-1/data/instance:
uuid: "ee795d4e32df45e0bae9948008b5be69"
format_stamp: "Formatted at 2026-06-19 18:51:41 on dist-test-slave-m9wg"
server_key: "d973b9db6754d2680e89aab86d92c825"
server_key_iv: "9d0e695f3cecef0d00f07ed77db1abd5"
server_key_version: "encryptionkey@0"
I20260619 18:51:41.138217 21940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894810715282-4045-0/minicluster-data/ts-1/wal/instance:
uuid: "ee795d4e32df45e0bae9948008b5be69"
format_stamp: "Formatted at 2026-06-19 18:51:41 on dist-test-slave-m9wg"
server_key: "d973b9db6754d2680e89aab86d92c825"
server_key_iv: "9d0e695f3cecef0d00f07ed77db1abd5"
server_key_version: "encryptionkey@0"
I20260619 18:51:41.142621 21940 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 18:51:41.145658 21954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:51:41.146791 21940 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 18:51:41.146929 21940 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894810715282-4045-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894810715282-4045-0/minicluster-data/ts-1/wal
uuid: "ee795d4e32df45e0bae9948008b5be69"
format_stamp: "Formatted at 2026-06-19 18:51:41 on dist-test-slave-m9wg"
server_key: "d973b9db6754d2680e89aab86d92c825"
server_key_iv: "9d0e695f3cecef0d00f07ed77db1abd5"
server_key_version: "encryptionkey@0"
I20260619 18:51:41.147107 21940 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894810715282-4045-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894810715282-4045-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894810715282-4045-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:51:41.181226 21940 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:51:41.182022 21940 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:51:41.182298 21940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:51:41.183228 21940 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:51:41.184607 21940 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:51:41.184701 21940 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:51:41.184811 21940 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:51:41.184885 21940 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:51:41.213308 21940 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.66:45601
I20260619 18:51:41.213371 22066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.66:45601 every 8 connection(s)
I20260619 18:51:41.214571 21940 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894810715282-4045-0/minicluster-data/ts-1/data/info.pb
I20260619 18:51:41.217320 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 21940
I20260619 18:51:41.217494 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894810715282-4045-0/minicluster-data/ts-1/wal/instance
I20260619 18:51:41.218150 4045 external_mini_cluster.cc:1428] Setting key f35993f14d7ef84224a3809247b8e20f
I20260619 18:51:41.220803 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894810715282-4045-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894810715282-4045-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894810715282-4045-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894810715282-4045-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.67:0
--local_ip_for_outbound_sockets=127.3.243.67
--webserver_interface=127.3.243.67
--webserver_port=0
--tserver_master_addrs=127.3.243.126:33449
--builtin_ntp_servers=127.3.243.84:43173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 18:51:41.226447 21938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:41.216887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.66:33019 (local address 127.3.243.126:33449)
0619 18:51:41.217043 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 18:51:41.217047 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:51:41.218645 (+ 1598us) server_negotiation.cc:407] Connection header received
0619 18:51:41.220546 (+ 1901us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:51:41.220552 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:51:41.220629 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:51:41.220713 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:51:41.221765 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:41.222462 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:41.223943 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:41.224379 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:41.224489 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:41.225701 (+ 1212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:51:41.225706 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:51:41.225709 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:51:41.225726 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:51:41.225756 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:51:41.225767 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:51:41.225852 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 18:51:41.226039 (+ 187us) server_negotiation.cc:299] Negotiation successful
0619 18:51:41.226138 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 18:51:41.227353 22069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:41.217557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:33449 (local address 127.3.243.66:33019)
0619 18:51:41.218497 (+ 940us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:41.218539 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 18:51:41.220293 (+ 1754us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:51:41.220966 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:51:41.220978 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:51:41.221069 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:51:41.221607 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:51:41.221624 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:51:41.222601 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:51:41.222605 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:51:41.223798 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:51:41.223814 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:51:41.224582 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:51:41.224589 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:51:41.224792 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:41.225425 (+ 633us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:51:41.225427 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:51:41.225462 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:51:41.225531 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:51:41.225851 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:51:41.225862 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:51:41.225870 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 18:51:41.226018 (+ 148us) client_negotiation.cc:240] Negotiation successful
0619 18:51:41.226370 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":291,"threads_started":1}
I20260619 18:51:41.228633 22067 heartbeater.cc:347] Connected to a master server at 127.3.243.126:33449
I20260619 18:51:41.228879 22067 heartbeater.cc:464] Registering TS with master...
I20260619 18:51:41.229521 22067 heartbeater.cc:511] Master 127.3.243.126:33449 requested a full tablet report, sending...
I20260619 18:51:41.230746 21750 ts_manager.cc:194] Registered new tserver with Master: ee795d4e32df45e0bae9948008b5be69 (127.3.243.66:45601)
I20260619 18:51:41.231602 21750 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.243.66:33019
W20260619 18:51:41.534047 22070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:51:41.534387 22070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:51:41.534529 22070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:51:41.541968 22070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:51:41.542068 22070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:51:41.542215 22070 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.67
I20260619 18:51:41.551290 22070 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:43173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894810715282-4045-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894810715282-4045-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894810715282-4045-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.243.67
--webserver_port=0
--tserver_master_addrs=127.3.243.126:33449
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.67
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894810715282-4045-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:51:41.552595 22070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:51:41.554337 22070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:51:41.566336 22076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:51:41.566433 22078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:51:41.566416 22075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:51:41.567303 22070 server_base.cc:1034] running on GCE node
I20260619 18:51:41.567845 22070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:51:41.568770 22070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:51:41.569964 22070 hybrid_clock.cc:648] HybridClock initialized: now 1781895101569912 us; error 43 us; skew 500 ppm
I20260619 18:51:41.570248 22070 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 18:51:41.572290 22070 webserver.cc:460] Webserver started at http://127.3.243.67:42101/ using document root <none> and password file <none>
I20260619 18:51:41.572935 22070 fs_manager.cc:359] Metadata directory not provided
I20260619 18:51:41.573031 22070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:51:41.573320 22070 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:51:41.575999 22070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894810715282-4045-0/minicluster-data/ts-2/data/instance:
uuid: "68132129b90e4084a0faa07c0078765c"
format_stamp: "Formatted at 2026-06-19 18:51:41 on dist-test-slave-m9wg"
server_key: "18a06641c26b185b73d6744a1c262e6b"
server_key_iv: "96f1853c7391202681cf03150f095555"
server_key_version: "encryptionkey@0"
I20260619 18:51:41.576786 22070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894810715282-4045-0/minicluster-data/ts-2/wal/instance:
uuid: "68132129b90e4084a0faa07c0078765c"
format_stamp: "Formatted at 2026-06-19 18:51:41 on dist-test-slave-m9wg"
server_key: "18a06641c26b185b73d6744a1c262e6b"
server_key_iv: "96f1853c7391202681cf03150f095555"
server_key_version: "encryptionkey@0"
I20260619 18:51:41.581485 22070 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260619 18:51:41.584650 22084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:51:41.585845 22070 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 18:51:41.585988 22070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894810715282-4045-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894810715282-4045-0/minicluster-data/ts-2/wal
uuid: "68132129b90e4084a0faa07c0078765c"
format_stamp: "Formatted at 2026-06-19 18:51:41 on dist-test-slave-m9wg"
server_key: "18a06641c26b185b73d6744a1c262e6b"
server_key_iv: "96f1853c7391202681cf03150f095555"
server_key_version: "encryptionkey@0"
I20260619 18:51:41.586174 22070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894810715282-4045-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894810715282-4045-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894810715282-4045-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:51:41.609057 22070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:51:41.609798 22070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:51:41.610069 22070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:51:41.610963 22070 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:51:41.612381 22070 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:51:41.612474 22070 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:51:41.612572 22070 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:51:41.612638 22070 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:51:41.641201 22070 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.67:41763
I20260619 18:51:41.641259 22196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.67:41763 every 8 connection(s)
I20260619 18:51:41.642421 22070 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894810715282-4045-0/minicluster-data/ts-2/data/info.pb
I20260619 18:51:41.651978 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 22070
I20260619 18:51:41.652175 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894810715282-4045-0/minicluster-data/ts-2/wal/instance
I20260619 18:51:41.652683 4045 external_mini_cluster.cc:1428] Setting key 328a4c6be841327159fc5e60360c0441
I20260619 18:51:41.654368 21938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:41.644722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.67:34845 (local address 127.3.243.126:33449)
0619 18:51:41.644928 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0619 18:51:41.644934 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:51:41.646237 (+ 1303us) server_negotiation.cc:407] Connection header received
0619 18:51:41.647947 (+ 1710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:51:41.647952 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:51:41.648020 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:51:41.648106 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:51:41.649186 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:41.650073 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:41.651574 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:41.652186 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:41.652335 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:41.653572 (+ 1237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:51:41.653577 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:51:41.653580 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:51:41.653597 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:51:41.653627 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:51:41.653638 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:51:41.653719 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 18:51:41.653949 (+ 230us) server_negotiation.cc:299] Negotiation successful
0619 18:51:41.654044 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 18:51:41.655306 22199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:41.645283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:33449 (local address 127.3.243.67:34845)
0619 18:51:41.646098 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:41.646137 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 18:51:41.647687 (+ 1550us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:51:41.648349 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:51:41.648360 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:51:41.648478 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:51:41.649019 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:51:41.649040 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:51:41.650200 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:51:41.650203 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:51:41.651449 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:51:41.651460 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:51:41.652374 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:51:41.652379 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:51:41.652567 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:41.653262 (+ 695us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:51:41.653265 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:51:41.653317 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:51:41.653416 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:51:41.653726 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:51:41.653736 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:51:41.653744 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 18:51:41.653920 (+ 176us) client_negotiation.cc:240] Negotiation successful
0619 18:51:41.654221 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":229,"threads_started":1}
I20260619 18:51:41.656414 22197 heartbeater.cc:347] Connected to a master server at 127.3.243.126:33449
I20260619 18:51:41.656665 22197 heartbeater.cc:464] Registering TS with master...
I20260619 18:51:41.657292 22197 heartbeater.cc:511] Master 127.3.243.126:33449 requested a full tablet report, sending...
I20260619 18:51:41.658505 21750 ts_manager.cc:194] Registered new tserver with Master: 68132129b90e4084a0faa07c0078765c (127.3.243.67:41763)
I20260619 18:51:41.659253 21750 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.243.67:34845
I20260619 18:51:41.667222 4045 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 18:51:41.675979 21938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:41.670988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47652 (local address 127.3.243.126:33449)
0619 18:51:41.671136 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 18:51:41.671140 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:51:41.671610 (+ 470us) server_negotiation.cc:407] Connection header received
0619 18:51:41.671737 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:51:41.671741 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:51:41.671806 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:51:41.671888 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:51:41.672283 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:41.672955 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:41.674181 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:41.674537 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:41.674617 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:41.675484 (+ 867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:51:41.675488 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:51:41.675491 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:51:41.675506 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:51:41.675530 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:51:41.675539 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:51:41.675598 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 18:51:41.675678 (+ 80us) server_negotiation.cc:299] Negotiation successful
0619 18:51:41.675742 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 18:51:41.679277 21750 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47652:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:51:41.685743 21750 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:51:41.719251 22211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:41.697289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56646 (local address 127.3.243.66:45601)
0619 18:51:41.710579 (+ 13290us) server_negotiation.cc:206] Beginning negotiation
0619 18:51:41.710589 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:51:41.710623 (+ 34us) server_negotiation.cc:407] Connection header received
0619 18:51:41.710734 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:51:41.710742 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:51:41.711467 (+ 725us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:51:41.711656 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:51:41.712500 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:41.713495 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:41.716129 (+ 2634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:41.717403 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:41.717906 (+ 503us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:41.718167 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0619 18:51:41.718690 (+ 523us) server_negotiation.cc:299] Negotiation successful
0619 18:51:41.718856 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13095,"thread_start_us":12970,"threads_started":1}
I20260619 18:51:41.719625 22209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:41.697109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.65:33439 (local address 127.0.0.1:56938)
0619 18:51:41.710896 (+ 13787us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:41.710914 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 18:51:41.711028 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:51:41.712338 (+ 1310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:51:41.712342 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:51:41.712367 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:51:41.712785 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:51:41.712798 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:51:41.715091 (+ 2293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:51:41.715096 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:51:41.717053 (+ 1957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:51:41.717063 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:51:41.718849 (+ 1786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:51:41.718855 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:51:41.719042 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:41.719077 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 18:51:41.719172 (+ 95us) client_negotiation.cc:240] Negotiation successful
0619 18:51:41.719305 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13651,"mutex_wait_us":3,"thread_start_us":14012,"threads_started":1}
I20260619 18:51:41.719985 22212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:41.697355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56938 (local address 127.3.243.65:33439)
0619 18:51:41.711741 (+ 14386us) server_negotiation.cc:206] Beginning negotiation
0619 18:51:41.711749 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:51:41.711777 (+ 28us) server_negotiation.cc:407] Connection header received
0619 18:51:41.711869 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:51:41.711877 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:51:41.712104 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:51:41.713003 (+ 899us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:51:41.713248 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:41.714951 (+ 1703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:41.717227 (+ 2276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:41.718561 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:41.718684 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:41.718926 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0619 18:51:41.719227 (+ 301us) server_negotiation.cc:299] Negotiation successful
0619 18:51:41.719628 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14210,"thread_start_us":14157,"threads_started":1}
I20260619 18:51:41.720405 22210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:41.696946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.66:45601 (local address 127.0.0.1:56646)
0619 18:51:41.709682 (+ 12736us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:41.709731 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 18:51:41.709937 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:51:41.711927 (+ 1990us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:51:41.711935 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:51:41.711968 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:51:41.712327 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:51:41.712342 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:51:41.713667 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:51:41.713672 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:51:41.715973 (+ 2301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:51:41.715987 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:51:41.718280 (+ 2293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:51:41.718286 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:51:41.718475 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:41.718520 (+ 45us) client_negotiation.cc:769] Sending connection context
0619 18:51:41.718626 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 18:51:41.720070 (+ 1444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12422,"mutex_wait_us":1266,"thread_start_us":13681,"threads_started":1}
I20260619 18:51:41.722930 22214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:41.712358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.67:41763 (local address 127.0.0.1:57588)
0619 18:51:41.715534 (+ 3176us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:41.715548 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 18:51:41.715667 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:51:41.716232 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:51:41.716237 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:51:41.716261 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:51:41.716614 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:51:41.716625 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:51:41.718359 (+ 1734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:51:41.718367 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:51:41.720335 (+ 1968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:51:41.720350 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:51:41.722297 (+ 1947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:51:41.722303 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:51:41.722428 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:41.722449 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 18:51:41.722515 (+ 66us) client_negotiation.cc:240] Negotiation successful
0619 18:51:41.722625 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3046,"mutex_wait_us":1,"thread_start_us":2887,"threads_started":1}
I20260619 18:51:41.723062 22213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:41.711718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57588 (local address 127.3.243.67:41763)
0619 18:51:41.712360 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0619 18:51:41.712367 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:51:41.715639 (+ 3272us) server_negotiation.cc:407] Connection header received
0619 18:51:41.715797 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:51:41.715805 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:51:41.715982 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:51:41.716189 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:51:41.716782 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:41.718213 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:41.720562 (+ 2349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:41.721809 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:41.721926 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:41.722183 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0619 18:51:41.722595 (+ 412us) server_negotiation.cc:299] Negotiation successful
0619 18:51:41.722751 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":286,"threads_started":1}
I20260619 18:51:41.723821 22002 tablet_service.cc:1467] Processing CreateTablet for tablet 75d7b0ea922845b69bad2a53b797186f (DEFAULT_TABLE table=test-table [id=d3c0259d763d4887a32af588de8bcce5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:51:41.723824 21871 tablet_service.cc:1467] Processing CreateTablet for tablet 75d7b0ea922845b69bad2a53b797186f (DEFAULT_TABLE table=test-table [id=d3c0259d763d4887a32af588de8bcce5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:51:41.725240 21871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75d7b0ea922845b69bad2a53b797186f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:51:41.725239 22002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75d7b0ea922845b69bad2a53b797186f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:51:41.727041 22132 tablet_service.cc:1467] Processing CreateTablet for tablet 75d7b0ea922845b69bad2a53b797186f (DEFAULT_TABLE table=test-table [id=d3c0259d763d4887a32af588de8bcce5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:51:41.728216 22132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75d7b0ea922845b69bad2a53b797186f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:51:41.738610 22215 tablet_bootstrap.cc:492] T 75d7b0ea922845b69bad2a53b797186f P 68132129b90e4084a0faa07c0078765c: Bootstrap starting.
I20260619 18:51:41.740226 22216 tablet_bootstrap.cc:492] T 75d7b0ea922845b69bad2a53b797186f P 61ad01072f68435aab5d63e0bf38e9cc: Bootstrap starting.
I20260619 18:51:41.741601 22217 tablet_bootstrap.cc:492] T 75d7b0ea922845b69bad2a53b797186f P ee795d4e32df45e0bae9948008b5be69: Bootstrap starting.
I20260619 18:51:41.743255 22215 tablet_bootstrap.cc:654] T 75d7b0ea922845b69bad2a53b797186f P 68132129b90e4084a0faa07c0078765c: Neither blocks nor log segments found. Creating new log.
I20260619 18:51:41.743361 22216 tablet_bootstrap.cc:654] T 75d7b0ea922845b69bad2a53b797186f P 61ad01072f68435aab5d63e0bf38e9cc: Neither blocks nor log segments found. Creating new log.
I20260619 18:51:41.744448 22217 tablet_bootstrap.cc:654] T 75d7b0ea922845b69bad2a53b797186f P ee795d4e32df45e0bae9948008b5be69: Neither blocks nor log segments found. Creating new log.
I20260619 18:51:41.744553 22216 log.cc:824] T 75d7b0ea922845b69bad2a53b797186f P 61ad01072f68435aab5d63e0bf38e9cc: Log is configured to *not* fsync() on all Append() calls
I20260619 18:51:41.744539 22215 log.cc:824] T 75d7b0ea922845b69bad2a53b797186f P 68132129b90e4084a0faa07c0078765c: Log is configured to *not* fsync() on all Append() calls
I20260619 18:51:41.745538 22217 log.cc:824] T 75d7b0ea922845b69bad2a53b797186f P ee795d4e32df45e0bae9948008b5be69: Log is configured to *not* fsync() on all Append() calls
I20260619 18:51:41.747424 22215 tablet_bootstrap.cc:492] T 75d7b0ea922845b69bad2a53b797186f P 68132129b90e4084a0faa07c0078765c: No bootstrap required, opened a new log
I20260619 18:51:41.747691 22215 ts_tablet_manager.cc:1397] T 75d7b0ea922845b69bad2a53b797186f P 68132129b90e4084a0faa07c0078765c: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260619 18:51:41.747774 22217 tablet_bootstrap.cc:492] T 75d7b0ea922845b69bad2a53b797186f P ee795d4e32df45e0bae9948008b5be69: No bootstrap required, opened a new log
I20260619 18:51:41.747994 22217 ts_tablet_manager.cc:1397] T 75d7b0ea922845b69bad2a53b797186f P ee795d4e32df45e0bae9948008b5be69: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260619 18:51:41.748461 22216 tablet_bootstrap.cc:492] T 75d7b0ea922845b69bad2a53b797186f P 61ad01072f68435aab5d63e0bf38e9cc: No bootstrap required, opened a new log
I20260619 18:51:41.748705 22216 ts_tablet_manager.cc:1397] T 75d7b0ea922845b69bad2a53b797186f P 61ad01072f68435aab5d63e0bf38e9cc: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260619 18:51:41.754576 22217 raft_consensus.cc:348] T 75d7b0ea922845b69bad2a53b797186f P ee795d4e32df45e0bae9948008b5be69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee795d4e32df45e0bae9948008b5be69" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 45601 } } peers { permanent_uuid: "61ad01072f68435aab5d63e0bf38e9cc" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 33439 } } peers { permanent_uuid: "68132129b90e4084a0faa07c0078765c" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 41763 } }
I20260619 18:51:41.754901 22217 raft_consensus.cc:374] T 75d7b0ea922845b69bad2a53b797186f P ee795d4e32df45e0bae9948008b5be69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:51:41.754992 22217 raft_consensus.cc:729] T 75d7b0ea922845b69bad2a53b797186f P ee795d4e32df45e0bae9948008b5be69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee795d4e32df45e0bae9948008b5be69, State: Initialized, Role: FOLLOWER
I20260619 18:51:41.755724 22217 consensus_queue.cc:260] T 75d7b0ea922845b69bad2a53b797186f P ee795d4e32df45e0bae9948008b5be69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee795d4e32df45e0bae9948008b5be69" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 45601 } } peers { permanent_uuid: "61ad01072f68435aab5d63e0bf38e9cc" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 33439 } } peers { permanent_uuid: "68132129b90e4084a0faa07c0078765c" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 41763 } }
I20260619 18:51:41.756680 22067 heartbeater.cc:503] Master 127.3.243.126:33449 was elected leader, sending a full tablet report...
I20260619 18:51:41.757277 22217 ts_tablet_manager.cc:1428] T 75d7b0ea922845b69bad2a53b797186f P ee795d4e32df45e0bae9948008b5be69: Time spent starting tablet: real 0.009s user 0.010s sys 0.000s
I20260619 18:51:41.757624 22215 raft_consensus.cc:348] T 75d7b0ea922845b69bad2a53b797186f P 68132129b90e4084a0faa07c0078765c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee795d4e32df45e0bae9948008b5be69" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 45601 } } peers { permanent_uuid: "61ad01072f68435aab5d63e0bf38e9cc" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 33439 } } peers { permanent_uuid: "68132129b90e4084a0faa07c0078765c" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 41763 } }
I20260619 18:51:41.758008 22215 raft_consensus.cc:374] T 75d7b0ea922845b69bad2a53b797186f P 68132129b90e4084a0faa07c0078765c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:51:41.758108 22215 raft_consensus.cc:729] T 75d7b0ea922845b69bad2a53b797186f P 68132129b90e4084a0faa07c0078765c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68132129b90e4084a0faa07c0078765c, State: Initialized, Role: FOLLOWER
I20260619 18:51:41.758663 22215 consensus_queue.cc:260] T 75d7b0ea922845b69bad2a53b797186f P 68132129b90e4084a0faa07c0078765c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee795d4e32df45e0bae9948008b5be69" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 45601 } } peers { permanent_uuid: "61ad01072f68435aab5d63e0bf38e9cc" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 33439 } } peers { permanent_uuid: "68132129b90e4084a0faa07c0078765c" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 41763 } }
I20260619 18:51:41.759625 22197 heartbeater.cc:503] Master 127.3.243.126:33449 was elected leader, sending a full tablet report...
I20260619 18:51:41.759873 22216 raft_consensus.cc:348] T 75d7b0ea922845b69bad2a53b797186f P 61ad01072f68435aab5d63e0bf38e9cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee795d4e32df45e0bae9948008b5be69" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 45601 } } peers { permanent_uuid: "61ad01072f68435aab5d63e0bf38e9cc" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 33439 } } peers { permanent_uuid: "68132129b90e4084a0faa07c0078765c" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 41763 } }
I20260619 18:51:41.760241 22215 ts_tablet_manager.cc:1428] T 75d7b0ea922845b69bad2a53b797186f P 68132129b90e4084a0faa07c0078765c: Time spent starting tablet: real 0.012s user 0.013s sys 0.002s
I20260619 18:51:41.760294 22216 raft_consensus.cc:374] T 75d7b0ea922845b69bad2a53b797186f P 61ad01072f68435aab5d63e0bf38e9cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:51:41.760386 22216 raft_consensus.cc:729] T 75d7b0ea922845b69bad2a53b797186f P 61ad01072f68435aab5d63e0bf38e9cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61ad01072f68435aab5d63e0bf38e9cc, State: Initialized, Role: FOLLOWER
I20260619 18:51:41.761541 22216 consensus_queue.cc:260] T 75d7b0ea922845b69bad2a53b797186f P 61ad01072f68435aab5d63e0bf38e9cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee795d4e32df45e0bae9948008b5be69" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 45601 } } peers { permanent_uuid: "61ad01072f68435aab5d63e0bf38e9cc" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 33439 } } peers { permanent_uuid: "68132129b90e4084a0faa07c0078765c" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 41763 } }
I20260619 18:51:41.762668 21936 heartbeater.cc:503] Master 127.3.243.126:33449 was elected leader, sending a full tablet report...
I20260619 18:51:41.763253 22216 ts_tablet_manager.cc:1428] T 75d7b0ea922845b69bad2a53b797186f P 61ad01072f68435aab5d63e0bf38e9cc: Time spent starting tablet: real 0.014s user 0.008s sys 0.004s
I20260619 18:51:41.779971 22221 raft_consensus.cc:482] T 75d7b0ea922845b69bad2a53b797186f P ee795d4e32df45e0bae9948008b5be69 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:51:41.780220 22221 raft_consensus.cc:504] T 75d7b0ea922845b69bad2a53b797186f P ee795d4e32df45e0bae9948008b5be69 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee795d4e32df45e0bae9948008b5be69" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 45601 } } peers { permanent_uuid: "61ad01072f68435aab5d63e0bf38e9cc" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 33439 } } peers { permanent_uuid: "68132129b90e4084a0faa07c0078765c" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 41763 } }
I20260619 18:51:41.781625 22221 leader_election.cc:288] T 75d7b0ea922845b69bad2a53b797186f P ee795d4e32df45e0bae9948008b5be69 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 61ad01072f68435aab5d63e0bf38e9cc (127.3.243.65:33439), 68132129b90e4084a0faa07c0078765c (127.3.243.67:41763)
I20260619 18:51:41.789845 22213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:41.781962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.66:40667 (local address 127.3.243.67:41763)
0619 18:51:41.782144 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 18:51:41.782149 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:51:41.782807 (+ 658us) server_negotiation.cc:407] Connection header received
0619 18:51:41.782998 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:51:41.783002 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:51:41.783066 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:51:41.783156 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:51:41.783882 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:41.784957 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:41.787682 (+ 2725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:41.788863 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:41.789006 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:41.789069 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 18:51:41.789499 (+ 430us) server_negotiation.cc:299] Negotiation successful
0619 18:51:41.789628 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 18:51:41.790144 22224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:41.781841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.67:41763 (local address 127.3.243.66:40667)
0619 18:51:41.782691 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:41.782710 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 18:51:41.782889 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:51:41.783179 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:51:41.783185 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:51:41.783217 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:51:41.783753 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:51:41.783762 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:51:41.785059 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:51:41.785065 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:51:41.787496 (+ 2431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:51:41.787512 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:51:41.789055 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:51:41.789062 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:51:41.789285 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:41.789317 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 18:51:41.789465 (+ 148us) client_negotiation.cc:240] Negotiation successful
0619 18:51:41.789595 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":681,"mutex_wait_us":176,"spinlock_wait_cycles":31360,"thread_start_us":450,"threads_started":1}
I20260619 18:51:41.790378 22212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:41.782040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.66:48531 (local address 127.3.243.65:33439)
0619 18:51:41.782194 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 18:51:41.782199 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:51:41.782822 (+ 623us) server_negotiation.cc:407] Connection header received
0619 18:51:41.782982 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:51:41.782987 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:51:41.783067 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:51:41.783151 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:51:41.783888 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:41.784945 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:41.788025 (+ 3080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:41.789017 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:41.789131 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:41.789194 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 18:51:41.789972 (+ 778us) server_negotiation.cc:299] Negotiation successful
0619 18:51:41.790091 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 18:51:41.791051 22225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:41.781950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.65:33439 (local address 127.3.243.66:48531)
0619 18:51:41.782735 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:41.782748 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 18:51:41.782854 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:51:41.783179 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:51:41.783185 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:51:41.783228 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:51:41.783727 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:51:41.783745 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:51:41.785061 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:51:41.785065 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:51:41.787815 (+ 2750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:51:41.787829 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:51:41.789597 (+ 1768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:51:41.789602 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:51:41.789775 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:41.789804 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 18:51:41.790608 (+ 804us) client_negotiation.cc:240] Negotiation successful
0619 18:51:41.790729 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":685,"mutex_wait_us":163,"spinlock_wait_cycles":3840,"thread_start_us":398,"threads_started":1}
W20260619 18:51:41.791025 21937 tablet.cc:2369] T 75d7b0ea922845b69bad2a53b797186f P 61ad01072f68435aab5d63e0bf38e9cc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:51:41.791142 22152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75d7b0ea922845b69bad2a53b797186f" candidate_uuid: "ee795d4e32df45e0bae9948008b5be69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68132129b90e4084a0faa07c0078765c" is_pre_election: true
I20260619 18:51:41.791653 22152 raft_consensus.cc:2454] T 75d7b0ea922845b69bad2a53b797186f P 68132129b90e4084a0faa07c0078765c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee795d4e32df45e0bae9948008b5be69 in term 0.
I20260619 18:51:41.791956 21891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75d7b0ea922845b69bad2a53b797186f" candidate_uuid: "ee795d4e32df45e0bae9948008b5be69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61ad01072f68435aab5d63e0bf38e9cc" is_pre_election: true
I20260619 18:51:41.792330 21956 leader_election.cc:302] T 75d7b0ea922845b69bad2a53b797186f P ee795d4e32df45e0bae9948008b5be69 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68132129b90e4084a0faa07c0078765c, ee795d4e32df45e0bae9948008b5be69; no voters:
I20260619 18:51:41.792500 21891 raft_consensus.cc:2454] T 75d7b0ea922845b69bad2a53b797186f P 61ad01072f68435aab5d63e0bf38e9cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee795d4e32df45e0bae9948008b5be69 in term 0.
I20260619 18:51:41.792955 22221 raft_consensus.cc:2781] T 75d7b0ea922845b69bad2a53b797186f P ee795d4e32df45e0bae9948008b5be69 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:51:41.793081 22221 raft_consensus.cc:482] T 75d7b0ea922845b69bad2a53b797186f P ee795d4e32df45e0bae9948008b5be69 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:51:41.793152 22221 raft_consensus.cc:3037] T 75d7b0ea922845b69bad2a53b797186f P ee795d4e32df45e0bae9948008b5be69 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:51:41.795222 22221 raft_consensus.cc:504] T 75d7b0ea922845b69bad2a53b797186f P ee795d4e32df45e0bae9948008b5be69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee795d4e32df45e0bae9948008b5be69" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 45601 } } peers { permanent_uuid: "61ad01072f68435aab5d63e0bf38e9cc" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 33439 } } peers { permanent_uuid: "68132129b90e4084a0faa07c0078765c" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 41763 } }
I20260619 18:51:41.795749 22221 leader_election.cc:288] T 75d7b0ea922845b69bad2a53b797186f P ee795d4e32df45e0bae9948008b5be69 [CANDIDATE]: Term 1 election: Requested vote from peers 61ad01072f68435aab5d63e0bf38e9cc (127.3.243.65:33439), 68132129b90e4084a0faa07c0078765c (127.3.243.67:41763)
I20260619 18:51:41.796223 21891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75d7b0ea922845b69bad2a53b797186f" candidate_uuid: "ee795d4e32df45e0bae9948008b5be69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61ad01072f68435aab5d63e0bf38e9cc"
I20260619 18:51:41.796214 22152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75d7b0ea922845b69bad2a53b797186f" candidate_uuid: "ee795d4e32df45e0bae9948008b5be69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68132129b90e4084a0faa07c0078765c"
I20260619 18:51:41.796413 21891 raft_consensus.cc:3037] T 75d7b0ea922845b69bad2a53b797186f P 61ad01072f68435aab5d63e0bf38e9cc [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:51:41.796417 22152 raft_consensus.cc:3037] T 75d7b0ea922845b69bad2a53b797186f P 68132129b90e4084a0faa07c0078765c [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:51:41.799422 22152 raft_consensus.cc:2454] T 75d7b0ea922845b69bad2a53b797186f P 68132129b90e4084a0faa07c0078765c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee795d4e32df45e0bae9948008b5be69 in term 1.
I20260619 18:51:41.799468 21891 raft_consensus.cc:2454] T 75d7b0ea922845b69bad2a53b797186f P 61ad01072f68435aab5d63e0bf38e9cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee795d4e32df45e0bae9948008b5be69 in term 1.
I20260619 18:51:41.799805 21957 leader_election.cc:302] T 75d7b0ea922845b69bad2a53b797186f P ee795d4e32df45e0bae9948008b5be69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61ad01072f68435aab5d63e0bf38e9cc, ee795d4e32df45e0bae9948008b5be69; no voters:
I20260619 18:51:41.800087 22221 raft_consensus.cc:2781] T 75d7b0ea922845b69bad2a53b797186f P ee795d4e32df45e0bae9948008b5be69 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:51:41.800510 22221 raft_consensus.cc:686] T 75d7b0ea922845b69bad2a53b797186f P ee795d4e32df45e0bae9948008b5be69 [term 1 LEADER]: Becoming Leader. State: Replica: ee795d4e32df45e0bae9948008b5be69, State: Running, Role: LEADER
I20260619 18:51:41.800917 22221 consensus_queue.cc:237] T 75d7b0ea922845b69bad2a53b797186f P ee795d4e32df45e0bae9948008b5be69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee795d4e32df45e0bae9948008b5be69" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 45601 } } peers { permanent_uuid: "61ad01072f68435aab5d63e0bf38e9cc" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 33439 } } peers { permanent_uuid: "68132129b90e4084a0faa07c0078765c" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 41763 } }
I20260619 18:51:41.805203 21750 catalog_manager.cc:5495] T 75d7b0ea922845b69bad2a53b797186f P ee795d4e32df45e0bae9948008b5be69 reported cstate change: term changed from 0 to 1, leader changed from <none> to ee795d4e32df45e0bae9948008b5be69 (127.3.243.66). New cstate: current_term: 1 leader_uuid: "ee795d4e32df45e0bae9948008b5be69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee795d4e32df45e0bae9948008b5be69" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 45601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61ad01072f68435aab5d63e0bf38e9cc" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 33439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "68132129b90e4084a0faa07c0078765c" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 41763 } health_report { overall_health: UNKNOWN } } }
I20260619 18:51:41.819414 22211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:41.814410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56658 (local address 127.3.243.66:45601)
0619 18:51:41.814560 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 18:51:41.814565 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:51:41.814604 (+ 39us) server_negotiation.cc:407] Connection header received
0619 18:51:41.814811 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:51:41.814815 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:51:41.814911 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:51:41.815062 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:51:41.815420 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:41.816091 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:41.817479 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:41.817881 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:41.817993 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:41.818464 (+ 471us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:51:41.818908 (+ 444us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:51:41.819034 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 18:51:41.819126 (+ 92us) server_negotiation.cc:299] Negotiation successful
0619 18:51:41.819199 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 18:51:41.826611 22152 raft_consensus.cc:1261] T 75d7b0ea922845b69bad2a53b797186f P 68132129b90e4084a0faa07c0078765c [term 1 FOLLOWER]: Refusing update from remote peer ee795d4e32df45e0bae9948008b5be69: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:51:41.826611 21891 raft_consensus.cc:1261] T 75d7b0ea922845b69bad2a53b797186f P 61ad01072f68435aab5d63e0bf38e9cc [term 1 FOLLOWER]: Refusing update from remote peer ee795d4e32df45e0bae9948008b5be69: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:51:41.827607 22221 consensus_queue.cc:1035] T 75d7b0ea922845b69bad2a53b797186f P ee795d4e32df45e0bae9948008b5be69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68132129b90e4084a0faa07c0078765c" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 41763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:51:41.827904 22226 consensus_queue.cc:1035] T 75d7b0ea922845b69bad2a53b797186f P ee795d4e32df45e0bae9948008b5be69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61ad01072f68435aab5d63e0bf38e9cc" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 33439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:51:41.840330 22231 mvcc.cc:205] Tried to move back new op lower bound from 7298642337072898048 to 7298642336980103168. Current Snapshot: MvccSnapshot[applied={T|T < 7298642337072898048}]
I20260619 18:51:41.840520 22230 mvcc.cc:205] Tried to move back new op lower bound from 7298642337072898048 to 7298642336980103168. Current Snapshot: MvccSnapshot[applied={T|T < 7298642337072898048}]
I20260619 18:51:41.854514 21749 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47652:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:51:41.854741 21749 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47652:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:51:41.858253 21749 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7e97bcf7ebbd4ba7a1d02c305fb5d186: Sending DeleteTablet for 3 replicas of tablet 75d7b0ea922845b69bad2a53b797186f
I20260619 18:51:41.859295 21871 tablet_service.cc:1514] Processing DeleteTablet for tablet 75d7b0ea922845b69bad2a53b797186f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:51:41 UTC) from {username='slave'} at 127.0.0.1:56938
I20260619 18:51:41.859622 22002 tablet_service.cc:1514] Processing DeleteTablet for tablet 75d7b0ea922845b69bad2a53b797186f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:51:41 UTC) from {username='slave'} at 127.0.0.1:56646
I20260619 18:51:41.859931 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 21809
I20260619 18:51:41.859964 22132 tablet_service.cc:1514] Processing DeleteTablet for tablet 75d7b0ea922845b69bad2a53b797186f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:51:41 UTC) from {username='slave'} at 127.0.0.1:57588
I20260619 18:51:41.863617 22239 tablet_replica.cc:330] T 75d7b0ea922845b69bad2a53b797186f P ee795d4e32df45e0bae9948008b5be69: stopping tablet replica
I20260619 18:51:41.864042 22239 raft_consensus.cc:2229] T 75d7b0ea922845b69bad2a53b797186f P ee795d4e32df45e0bae9948008b5be69 [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:51:41.864734 22239 raft_consensus.cc:2258] T 75d7b0ea922845b69bad2a53b797186f P ee795d4e32df45e0bae9948008b5be69 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:51:41.866734 22239 ts_tablet_manager.cc:1905] T 75d7b0ea922845b69bad2a53b797186f P ee795d4e32df45e0bae9948008b5be69: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:51:41.870134 22212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:41.861153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56948 (local address 127.3.243.65:33439)
0619 18:51:41.861429 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0619 18:51:41.861435 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:51:41.861738 (+ 303us) server_negotiation.cc:407] Connection header received
0619 18:51:41.862012 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:51:41.862018 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:51:41.862119 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:51:41.862188 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:51:41.862848 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:41.863961 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:41.866175 (+ 2214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:41.866845 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:41.866946 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:41.868477 (+ 1531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:51:41.868488 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:51:41.868503 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:51:41.869223 (+ 720us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:51:41.869279 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:51:41.869304 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:51:41.869385 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 18:51:41.869646 (+ 261us) server_negotiation.cc:299] Negotiation successful
0619 18:51:41.869783 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131}
I20260619 18:51:41.871392 21931 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:51:41.876700 22240 tablet_replica.cc:330] T 75d7b0ea922845b69bad2a53b797186f P 68132129b90e4084a0faa07c0078765c: stopping tablet replica
I20260619 18:51:41.877120 22240 raft_consensus.cc:2229] T 75d7b0ea922845b69bad2a53b797186f P 68132129b90e4084a0faa07c0078765c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:51:41.877633 22240 raft_consensus.cc:2258] T 75d7b0ea922845b69bad2a53b797186f P 68132129b90e4084a0faa07c0078765c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:51:41.877739 22239 ts_tablet_manager.cc:1918] T 75d7b0ea922845b69bad2a53b797186f P ee795d4e32df45e0bae9948008b5be69: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:51:41.877903 22239 log.cc:1196] T 75d7b0ea922845b69bad2a53b797186f P ee795d4e32df45e0bae9948008b5be69: Deleting WAL directory at /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894810715282-4045-0/minicluster-data/ts-1/wal/wals/75d7b0ea922845b69bad2a53b797186f
I20260619 18:51:41.878312 22239 ts_tablet_manager.cc:1939] T 75d7b0ea922845b69bad2a53b797186f P ee795d4e32df45e0bae9948008b5be69: Deleting consensus metadata
I20260619 18:51:41.879676 21735 catalog_manager.cc:4841] TS ee795d4e32df45e0bae9948008b5be69 (127.3.243.66:45601): tablet 75d7b0ea922845b69bad2a53b797186f (table test-table [id=d3c0259d763d4887a32af588de8bcce5]) successfully deleted
I20260619 18:51:41.881493 22240 ts_tablet_manager.cc:1905] T 75d7b0ea922845b69bad2a53b797186f P 68132129b90e4084a0faa07c0078765c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:51:41.885975 22240 ts_tablet_manager.cc:1918] T 75d7b0ea922845b69bad2a53b797186f P 68132129b90e4084a0faa07c0078765c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:51:41.886104 22240 log.cc:1196] T 75d7b0ea922845b69bad2a53b797186f P 68132129b90e4084a0faa07c0078765c: Deleting WAL directory at /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894810715282-4045-0/minicluster-data/ts-2/wal/wals/75d7b0ea922845b69bad2a53b797186f
I20260619 18:51:41.886509 22240 ts_tablet_manager.cc:1939] T 75d7b0ea922845b69bad2a53b797186f P 68132129b90e4084a0faa07c0078765c: Deleting consensus metadata
I20260619 18:51:41.887912 21735 catalog_manager.cc:4841] TS 68132129b90e4084a0faa07c0078765c (127.3.243.67:41763): tablet 75d7b0ea922845b69bad2a53b797186f (table test-table [id=d3c0259d763d4887a32af588de8bcce5]) successfully deleted
I20260619 18:51:41.975379 22238 tablet_replica.cc:330] T 75d7b0ea922845b69bad2a53b797186f P 61ad01072f68435aab5d63e0bf38e9cc: stopping tablet replica
I20260619 18:51:41.975694 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 21809
W20260619 18:51:41.996477 21736 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.243.65:33439 (error 108)
I20260619 18:51:41.996937 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 21940
I20260619 18:51:41.998602 22214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:41.998100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.65:33439 (local address 127.0.0.1:56964)
0619 18:51:41.998312 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:41.998437 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.65:33439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
W20260619 18:51:41.998857 21736 catalog_manager.cc:4584] TS 61ad01072f68435aab5d63e0bf38e9cc (127.3.243.65:33439): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 75d7b0ea922845b69bad2a53b797186f: Network error: Client connection negotiation failed: client connection to 127.3.243.65:33439: connect: Connection refused (error 111)
I20260619 18:51:42.004801 22211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:41.997690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56668 (local address 127.3.243.66:45601)
0619 18:51:41.997883 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 18:51:41.997889 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:51:41.997939 (+ 50us) server_negotiation.cc:407] Connection header received
0619 18:51:41.998154 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:51:41.998160 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:51:41.998274 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:51:41.998385 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:51:41.998891 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:41.999771 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:42.001668 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:42.002097 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:42.002191 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:42.003295 (+ 1104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:51:42.003304 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:51:42.003318 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:51:42.003893 (+ 575us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:51:42.003941 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:51:42.003961 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:51:42.004040 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 18:51:42.004393 (+ 353us) server_negotiation.cc:299] Negotiation successful
0619 18:51:42.004476 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 18:51:42.005647 22062 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:51:42.052560 22214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:42.052080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.65:33439 (local address 127.0.0.1:56972)
0619 18:51:42.052305 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:42.052394 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.65:33439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260619 18:51:42.053476 22214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:42.053163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.65:33439 (local address 127.0.0.1:56984)
0619 18:51:42.053293 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:42.053361 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.65:33439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260619 18:51:42.114017 22214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:42.113422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.65:33439 (local address 127.0.0.1:56988)
0619 18:51:42.113759 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:42.113856 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.65:33439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197}
I20260619 18:51:42.114886 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 21940
I20260619 18:51:42.115087 22214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:42.114712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.65:33439 (local address 127.0.0.1:56998)
0619 18:51:42.114864 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:42.114945 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.65:33439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260619 18:51:42.129820 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 22070
I20260619 18:51:42.135846 22213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:42.130562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57604 (local address 127.3.243.67:41763)
0619 18:51:42.130740 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 18:51:42.130747 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:51:42.130768 (+ 21us) server_negotiation.cc:407] Connection header received
0619 18:51:42.130858 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:51:42.130862 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:51:42.130959 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:51:42.131066 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:51:42.131383 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:42.132081 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:42.133344 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:42.133767 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:42.133855 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:42.134726 (+ 871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:51:42.134735 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:51:42.134748 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:51:42.135299 (+ 551us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:51:42.135344 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:51:42.135372 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:51:42.135458 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 18:51:42.135529 (+ 71us) server_negotiation.cc:299] Negotiation successful
0619 18:51:42.135611 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 18:51:42.136478 22192 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:51:42.195791 22214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:42.195324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.65:33439 (local address 127.0.0.1:57012)
0619 18:51:42.195537 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:42.195630 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.65:33439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260619 18:51:42.196859 22214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:42.196479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.65:33439 (local address 127.0.0.1:57026)
0619 18:51:42.196668 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:42.196738 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.65:33439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260619 18:51:42.222652 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 22070
I20260619 18:51:42.235893 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 21719
I20260619 18:51:42.236935 21780 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:51:42.342291 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 21719
2026-06-19T18:51:42Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2544 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894810715282-4045-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:51:42 dist-test-slave-m9wg krb5kdc[22249](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:51:42 dist-test-slave-m9wg krb5kdc[22249](info): set up 2 sockets
Jun 19 18:51:42 dist-test-slave-m9wg krb5kdc[22249](info): commencing operation
krb5kdc: starting...
W20260619 18:51:43.364094 4045 mini_kdc.cc:121] Time spent starting KDC: real 0.986s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894810715282-4045-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894810715282-4045-0/krb5kdc/test-user.keytab.
Jun 19 18:51:43 dist-test-slave-m9wg krb5kdc[22249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895103, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:51:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:51:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.243.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab.
Entry for principal kudu/127.3.243.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab.
I20260619 18:51:43.573360 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894810715282-4045-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894810715282-4045-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894810715282-4045-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.126:36197
--webserver_interface=127.3.243.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab
--principal=kudu/127.3.243.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:36469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.126:36197
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:51:43.881660 22262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:51:43.881960 22262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:51:43.882102 22262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:51:43.889183 22262 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:51:43.889292 22262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:51:43.889369 22262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:51:43.889451 22262 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:51:43.889513 22262 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:51:43.899111 22262 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:36469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894810715282-4045-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894810715282-4045-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.243.126:36197
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab
--principal=kudu/127.3.243.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.126:36197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.243.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894810715282-4045-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:51:43.900369 22262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:51:43.901664 22262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:51:43.912618 22268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:51:43.912694 22267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:51:43.912732 22270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:51:43.913226 22262 server_base.cc:1034] running on GCE node
I20260619 18:51:43.913932 22262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:51:43.915239 22262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:51:43.916437 22262 hybrid_clock.cc:648] HybridClock initialized: now 1781895103916391 us; error 30 us; skew 500 ppm
I20260619 18:51:43.916769 22262 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:51:43 dist-test-slave-m9wg krb5kdc[22249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895103, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:51:43.919750 22262 init.cc:377] Logged in from keytab as kudu/127.3.243.126@KRBTEST.COM (short username kudu)
I20260619 18:51:43.921190 22262 webserver.cc:460] Webserver started at http://127.3.243.126:42873/ using document root <none> and password file <none>
I20260619 18:51:43.921835 22262 fs_manager.cc:359] Metadata directory not provided
I20260619 18:51:43.921947 22262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:51:43.922222 22262 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:51:43.925283 22262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894810715282-4045-0/minicluster-data/master-0/data/instance:
uuid: "00c2537b1903461d9ba9f07c5e0b9e3c"
format_stamp: "Formatted at 2026-06-19 18:51:43 on dist-test-slave-m9wg"
server_key: "5b7060315af76cf3c78580b0be4762da"
server_key_iv: "44a90375d1f6a06658e4d46a85a4d56d"
server_key_version: "encryptionkey@0"
I20260619 18:51:43.925988 22262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894810715282-4045-0/minicluster-data/master-0/wal/instance:
uuid: "00c2537b1903461d9ba9f07c5e0b9e3c"
format_stamp: "Formatted at 2026-06-19 18:51:43 on dist-test-slave-m9wg"
server_key: "5b7060315af76cf3c78580b0be4762da"
server_key_iv: "44a90375d1f6a06658e4d46a85a4d56d"
server_key_version: "encryptionkey@0"
I20260619 18:51:43.930768 22262 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 18:51:43.933928 22277 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:51:43.935113 22262 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 18:51:43.935271 22262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894810715282-4045-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894810715282-4045-0/minicluster-data/master-0/wal
uuid: "00c2537b1903461d9ba9f07c5e0b9e3c"
format_stamp: "Formatted at 2026-06-19 18:51:43 on dist-test-slave-m9wg"
server_key: "5b7060315af76cf3c78580b0be4762da"
server_key_iv: "44a90375d1f6a06658e4d46a85a4d56d"
server_key_version: "encryptionkey@0"
I20260619 18:51:43.935431 22262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894810715282-4045-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894810715282-4045-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894810715282-4045-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:51:43.951911 22262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:51:43.955456 22262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:51:43.955706 22262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:51:43.971992 22262 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.126:36197
I20260619 18:51:43.972021 22328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.126:36197 every 8 connection(s)
I20260619 18:51:43.973268 22262 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
I20260619 18:51:43.976742 22329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:51:43.981803 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 22262
I20260619 18:51:43.981961 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894810715282-4045-0/minicluster-data/master-0/wal/instance
I20260619 18:51:43.982496 4045 external_mini_cluster.cc:1428] Setting key 715a4a1b70dd46d9edafaa9a946d48f0
I20260619 18:51:43.987144 22329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00c2537b1903461d9ba9f07c5e0b9e3c: Bootstrap starting.
I20260619 18:51:43.989976 22329 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 00c2537b1903461d9ba9f07c5e0b9e3c: Neither blocks nor log segments found. Creating new log.
I20260619 18:51:43.990955 22329 log.cc:824] T 00000000000000000000000000000000 P 00c2537b1903461d9ba9f07c5e0b9e3c: Log is configured to *not* fsync() on all Append() calls
I20260619 18:51:43.993498 22329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00c2537b1903461d9ba9f07c5e0b9e3c: No bootstrap required, opened a new log
Jun 19 18:51:43 dist-test-slave-m9wg krb5kdc[22249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895103, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.126@KRBTEST.COM
I20260619 18:51:44.000859 22329 raft_consensus.cc:348] T 00000000000000000000000000000000 P 00c2537b1903461d9ba9f07c5e0b9e3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00c2537b1903461d9ba9f07c5e0b9e3c" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 36197 } }
I20260619 18:51:44.001194 22329 raft_consensus.cc:374] T 00000000000000000000000000000000 P 00c2537b1903461d9ba9f07c5e0b9e3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:51:44.001293 22329 raft_consensus.cc:729] T 00000000000000000000000000000000 P 00c2537b1903461d9ba9f07c5e0b9e3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c2537b1903461d9ba9f07c5e0b9e3c, State: Initialized, Role: FOLLOWER
I20260619 18:51:44.001937 22329 consensus_queue.cc:260] T 00000000000000000000000000000000 P 00c2537b1903461d9ba9f07c5e0b9e3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00c2537b1903461d9ba9f07c5e0b9e3c" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 36197 } }
I20260619 18:51:44.002207 22329 raft_consensus.cc:388] T 00000000000000000000000000000000 P 00c2537b1903461d9ba9f07c5e0b9e3c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:51:44.002341 22329 raft_consensus.cc:482] T 00000000000000000000000000000000 P 00c2537b1903461d9ba9f07c5e0b9e3c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:51:44.002516 22329 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 00c2537b1903461d9ba9f07c5e0b9e3c [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:51:44.003346 22332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:43.984240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38212 (local address 127.3.243.126:36197)
0619 18:51:43.984990 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0619 18:51:43.985003 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 18:51:43.985043 (+ 40us) server_negotiation.cc:407] Connection header received
0619 18:51:43.986681 (+ 1638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:51:43.986703 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:51:43.987057 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:51:43.987407 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:51:43.988847 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:43.990065 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:43.991415 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:43.992081 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:43.992273 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:43.996024 (+ 3751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:51:43.996031 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:51:43.996045 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:51:43.996085 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:51:43.999085 (+ 3000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:51:43.999776 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:51:43.999783 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:51:43.999788 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:51:43.999965 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:51:44.000360 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:51:44.000364 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:51:44.000366 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:51:44.000890 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:51:44.001184 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0619 18:51:44.001614 (+ 430us) server_negotiation.cc:299] Negotiation successful
0619 18:51:44.002020 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":340,"threads_started":1}
I20260619 18:51:44.005030 22329 raft_consensus.cc:504] T 00000000000000000000000000000000 P 00c2537b1903461d9ba9f07c5e0b9e3c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00c2537b1903461d9ba9f07c5e0b9e3c" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 36197 } }
I20260619 18:51:44.005474 22329 leader_election.cc:302] T 00000000000000000000000000000000 P 00c2537b1903461d9ba9f07c5e0b9e3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 00c2537b1903461d9ba9f07c5e0b9e3c; no voters:
I20260619 18:51:44.006031 22329 leader_election.cc:288] T 00000000000000000000000000000000 P 00c2537b1903461d9ba9f07c5e0b9e3c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:51:44.006124 22334 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 00c2537b1903461d9ba9f07c5e0b9e3c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:51:44.006704 22334 raft_consensus.cc:686] T 00000000000000000000000000000000 P 00c2537b1903461d9ba9f07c5e0b9e3c [term 1 LEADER]: Becoming Leader. State: Replica: 00c2537b1903461d9ba9f07c5e0b9e3c, State: Running, Role: LEADER
I20260619 18:51:44.007229 22334 consensus_queue.cc:237] T 00000000000000000000000000000000 P 00c2537b1903461d9ba9f07c5e0b9e3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00c2537b1903461d9ba9f07c5e0b9e3c" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 36197 } }
I20260619 18:51:44.008014 22329 sys_catalog.cc:564] T 00000000000000000000000000000000 P 00c2537b1903461d9ba9f07c5e0b9e3c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:51:44.009778 22336 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00c2537b1903461d9ba9f07c5e0b9e3c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 00c2537b1903461d9ba9f07c5e0b9e3c. Latest consensus state: current_term: 1 leader_uuid: "00c2537b1903461d9ba9f07c5e0b9e3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00c2537b1903461d9ba9f07c5e0b9e3c" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 36197 } } }
I20260619 18:51:44.010788 22336 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00c2537b1903461d9ba9f07c5e0b9e3c [sys.catalog]: This master's current role is: LEADER
I20260619 18:51:44.010255 22335 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00c2537b1903461d9ba9f07c5e0b9e3c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "00c2537b1903461d9ba9f07c5e0b9e3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00c2537b1903461d9ba9f07c5e0b9e3c" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 36197 } } }
I20260619 18:51:44.010926 22335 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00c2537b1903461d9ba9f07c5e0b9e3c [sys.catalog]: This master's current role is: LEADER
I20260619 18:51:44.011662 22342 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:51:44.017030 22342 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:51:44.024938 22342 catalog_manager.cc:1347] Generated new cluster ID: cce5aa2631b8423e8193f175603d7290
I20260619 18:51:44.025043 22342 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:51:44.051709 22342 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:51:44.053658 22342 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:51:44.064337 22342 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 00c2537b1903461d9ba9f07c5e0b9e3c: Generated new TSK 0
I20260619 18:51:44.065382 22342 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.243.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.65.keytab.
Entry for principal kudu/127.3.243.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.65.keytab.
I20260619 18:51:44.126721 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894810715282-4045-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894810715282-4045-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894810715282-4045-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894810715282-4045-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.65:0
--local_ip_for_outbound_sockets=127.3.243.65
--webserver_interface=127.3.243.65
--webserver_port=0
--tserver_master_addrs=127.3.243.126:36197
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.65.keytab
--principal=kudu/127.3.243.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:36469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:51:44.436817 22355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:51:44.437076 22355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:51:44.437204 22355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:51:44.444255 22355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:51:44.444386 22355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:51:44.444532 22355 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.65
I20260619 18:51:44.454090 22355 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:36469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894810715282-4045-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894810715282-4045-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.65.keytab
--principal=kudu/127.3.243.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894810715282-4045-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.243.65
--webserver_port=0
--tserver_master_addrs=127.3.243.126:36197
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.65
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894810715282-4045-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:51:44.455315 22355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:51:44.456600 22355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:51:44.468736 22363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:51:44.468835 22361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:51:44.468822 22360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:51:44.469632 22355 server_base.cc:1034] running on GCE node
I20260619 18:51:44.470160 22355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:51:44.471009 22355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:51:44.472213 22355 hybrid_clock.cc:648] HybridClock initialized: now 1781895104472164 us; error 40 us; skew 500 ppm
I20260619 18:51:44.472504 22355 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:51:44 dist-test-slave-m9wg krb5kdc[22249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895104, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:51:44.475350 22355 init.cc:377] Logged in from keytab as kudu/127.3.243.65@KRBTEST.COM (short username kudu)
I20260619 18:51:44.476619 22355 webserver.cc:460] Webserver started at http://127.3.243.65:40167/ using document root <none> and password file <none>
I20260619 18:51:44.477201 22355 fs_manager.cc:359] Metadata directory not provided
I20260619 18:51:44.477294 22355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:51:44.477543 22355 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:51:44.480211 22355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894810715282-4045-0/minicluster-data/ts-0/data/instance:
uuid: "fb235187b06a41c7aa323544a09a5a10"
format_stamp: "Formatted at 2026-06-19 18:51:44 on dist-test-slave-m9wg"
server_key: "e3ab3d47616392921dcc78e804be5374"
server_key_iv: "ead5338f791586253c775a024d202ae7"
server_key_version: "encryptionkey@0"
I20260619 18:51:44.480928 22355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894810715282-4045-0/minicluster-data/ts-0/wal/instance:
uuid: "fb235187b06a41c7aa323544a09a5a10"
format_stamp: "Formatted at 2026-06-19 18:51:44 on dist-test-slave-m9wg"
server_key: "e3ab3d47616392921dcc78e804be5374"
server_key_iv: "ead5338f791586253c775a024d202ae7"
server_key_version: "encryptionkey@0"
I20260619 18:51:44.485453 22355 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 18:51:44.488452 22370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:51:44.489583 22355 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 18:51:44.489699 22355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894810715282-4045-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894810715282-4045-0/minicluster-data/ts-0/wal
uuid: "fb235187b06a41c7aa323544a09a5a10"
format_stamp: "Formatted at 2026-06-19 18:51:44 on dist-test-slave-m9wg"
server_key: "e3ab3d47616392921dcc78e804be5374"
server_key_iv: "ead5338f791586253c775a024d202ae7"
server_key_version: "encryptionkey@0"
I20260619 18:51:44.489852 22355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894810715282-4045-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894810715282-4045-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894810715282-4045-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:51:44.501951 22355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:51:44.505306 22355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:51:44.505556 22355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:51:44.506374 22355 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:51:44.507540 22355 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:51:44.507642 22355 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:51:44.507752 22355 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:51:44.507813 22355 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:51:44.535686 22355 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.65:35817
I20260619 18:51:44.535723 22482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.65:35817 every 8 connection(s)
I20260619 18:51:44.536880 22355 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894810715282-4045-0/minicluster-data/ts-0/data/info.pb
I20260619 18:51:44.541052 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 22355
I20260619 18:51:44.541224 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894810715282-4045-0/minicluster-data/ts-0/wal/instance
I20260619 18:51:44.541622 4045 external_mini_cluster.cc:1428] Setting key c981176d4b49b8b837e652c22e94795e
Jun 19 18:51:44 dist-test-slave-m9wg krb5kdc[22249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895104, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.65@KRBTEST.COM for kudu/127.3.243.126@KRBTEST.COM
I20260619 18:51:44.555770 22485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:44.539268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.65:42855 (local address 127.3.243.126:36197)
0619 18:51:44.539711 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0619 18:51:44.539716 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:51:44.540644 (+ 928us) server_negotiation.cc:407] Connection header received
0619 18:51:44.542531 (+ 1887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:51:44.542536 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:51:44.542618 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:51:44.542734 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:51:44.544234 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:44.544938 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:44.546419 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:44.546814 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:44.546921 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:44.550293 (+ 3372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:51:44.550298 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:51:44.550302 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:51:44.550335 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:51:44.552943 (+ 2608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:51:44.553726 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:51:44.553730 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:51:44.553732 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:51:44.553857 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:51:44.554273 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:51:44.554277 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:51:44.554279 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:51:44.554552 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:51:44.554659 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 18:51:44.555363 (+ 704us) server_negotiation.cc:299] Negotiation successful
0619 18:51:44.555519 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":248,"threads_started":1}
I20260619 18:51:44.557116 22486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:44.539805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:36197 (local address 127.3.243.65:42855)
0619 18:51:44.540490 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:44.540527 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 18:51:44.542230 (+ 1703us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:51:44.542915 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:51:44.542923 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:51:44.543498 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:51:44.544049 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:51:44.544066 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:51:44.545080 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:51:44.545084 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:51:44.546278 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:51:44.546290 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:51:44.547022 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:51:44.547029 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:51:44.547240 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:44.547905 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:51:44.547926 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:51:44.550063 (+ 2137us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:51:44.553081 (+ 3018us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:51:44.553088 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:51:44.553103 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:51:44.553630 (+ 527us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:51:44.553944 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:51:44.553948 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:51:44.553951 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:51:44.554185 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:51:44.554661 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:51:44.554669 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:51:44.555048 (+ 379us) client_negotiation.cc:769] Sending connection context
0619 18:51:44.555309 (+ 261us) client_negotiation.cc:240] Negotiation successful
0619 18:51:44.555784 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":266,"threads_started":1}
I20260619 18:51:44.558527 22483 heartbeater.cc:347] Connected to a master server at 127.3.243.126:36197
I20260619 18:51:44.558815 22483 heartbeater.cc:464] Registering TS with master...
I20260619 18:51:44.559465 22483 heartbeater.cc:511] Master 127.3.243.126:36197 requested a full tablet report, sending...
I20260619 18:51:44.561384 22294 ts_manager.cc:194] Registered new tserver with Master: fb235187b06a41c7aa323544a09a5a10 (127.3.243.65:35817)
I20260619 18:51:44.562778 22294 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.65@KRBTEST.COM'} at 127.3.243.65:42855
WARNING: no policy specified for kudu/127.3.243.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.66.keytab.
Entry for principal kudu/127.3.243.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.66.keytab.
I20260619 18:51:44.589411 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894810715282-4045-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894810715282-4045-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894810715282-4045-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894810715282-4045-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.66:0
--local_ip_for_outbound_sockets=127.3.243.66
--webserver_interface=127.3.243.66
--webserver_port=0
--tserver_master_addrs=127.3.243.126:36197
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.66.keytab
--principal=kudu/127.3.243.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:36469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:51:44.901178 22489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:51:44.901441 22489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:51:44.901538 22489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:51:44.908578 22489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:51:44.908689 22489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:51:44.908895 22489 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.66
I20260619 18:51:44.918362 22489 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:36469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894810715282-4045-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894810715282-4045-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.66.keytab
--principal=kudu/127.3.243.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894810715282-4045-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.243.66
--webserver_port=0
--tserver_master_addrs=127.3.243.126:36197
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.66
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894810715282-4045-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:51:44.919607 22489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:51:44.920910 22489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:51:44.932974 22497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:51:44.933014 22495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:51:44.933035 22494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:51:44.934023 22489 server_base.cc:1034] running on GCE node
I20260619 18:51:44.934623 22489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:51:44.935479 22489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:51:44.936669 22489 hybrid_clock.cc:648] HybridClock initialized: now 1781895104936622 us; error 35 us; skew 500 ppm
I20260619 18:51:44.936956 22489 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:51:44 dist-test-slave-m9wg krb5kdc[22249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895104, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:51:44.939947 22489 init.cc:377] Logged in from keytab as kudu/127.3.243.66@KRBTEST.COM (short username kudu)
I20260619 18:51:44.941352 22489 webserver.cc:460] Webserver started at http://127.3.243.66:46123/ using document root <none> and password file <none>
I20260619 18:51:44.942001 22489 fs_manager.cc:359] Metadata directory not provided
I20260619 18:51:44.942117 22489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:51:44.942411 22489 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:51:44.945204 22489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894810715282-4045-0/minicluster-data/ts-1/data/instance:
uuid: "70d7d7741d814dddbac57668dbc8b6e1"
format_stamp: "Formatted at 2026-06-19 18:51:44 on dist-test-slave-m9wg"
server_key: "75a2c2a9502546f343f4c448a8ddbe0d"
server_key_iv: "12f5978964919bb14c50183deddf8994"
server_key_version: "encryptionkey@0"
I20260619 18:51:44.945953 22489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894810715282-4045-0/minicluster-data/ts-1/wal/instance:
uuid: "70d7d7741d814dddbac57668dbc8b6e1"
format_stamp: "Formatted at 2026-06-19 18:51:44 on dist-test-slave-m9wg"
server_key: "75a2c2a9502546f343f4c448a8ddbe0d"
server_key_iv: "12f5978964919bb14c50183deddf8994"
server_key_version: "encryptionkey@0"
I20260619 18:51:44.950834 22489 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 18:51:44.954005 22504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:51:44.955210 22489 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 18:51:44.955354 22489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894810715282-4045-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894810715282-4045-0/minicluster-data/ts-1/wal
uuid: "70d7d7741d814dddbac57668dbc8b6e1"
format_stamp: "Formatted at 2026-06-19 18:51:44 on dist-test-slave-m9wg"
server_key: "75a2c2a9502546f343f4c448a8ddbe0d"
server_key_iv: "12f5978964919bb14c50183deddf8994"
server_key_version: "encryptionkey@0"
I20260619 18:51:44.955531 22489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894810715282-4045-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894810715282-4045-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894810715282-4045-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:51:44.975219 22489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:51:44.979045 22489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:51:44.979291 22489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:51:44.980254 22489 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:51:44.981482 22489 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:51:44.981554 22489 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:51:44.981670 22489 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:51:44.981719 22489 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:51:45.010340 22489 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.66:44645
I20260619 18:51:45.010386 22616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.66:44645 every 8 connection(s)
I20260619 18:51:45.011724 22489 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894810715282-4045-0/minicluster-data/ts-1/data/info.pb
I20260619 18:51:45.020411 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 22489
I20260619 18:51:45.020558 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894810715282-4045-0/minicluster-data/ts-1/wal/instance
I20260619 18:51:45.020978 4045 external_mini_cluster.cc:1428] Setting key 5f88e8837a0f6cd969deee6282f79427
Jun 19 18:51:45 dist-test-slave-m9wg krb5kdc[22249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895104, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.66@KRBTEST.COM for kudu/127.3.243.126@KRBTEST.COM
I20260619 18:51:45.030664 22485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:45.014264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.66:49421 (local address 127.3.243.126:36197)
0619 18:51:45.014444 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 18:51:45.014448 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:51:45.015561 (+ 1113us) server_negotiation.cc:407] Connection header received
0619 18:51:45.017400 (+ 1839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:51:45.017406 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:51:45.017491 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:51:45.017578 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:51:45.019045 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:45.019739 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:45.021345 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:45.021827 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:45.021955 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:45.025459 (+ 3504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:51:45.025464 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:51:45.025467 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:51:45.025498 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:51:45.027877 (+ 2379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:51:45.028589 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:51:45.028593 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:51:45.028595 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:51:45.028719 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:51:45.029098 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:51:45.029101 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:51:45.029103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:51:45.029337 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:51:45.029464 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 18:51:45.030103 (+ 639us) server_negotiation.cc:299] Negotiation successful
0619 18:51:45.030315 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 18:51:45.031522 22619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:45.014726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:36197 (local address 127.3.243.66:49421)
0619 18:51:45.015400 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:45.015438 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 18:51:45.017122 (+ 1684us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:51:45.017819 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:51:45.017827 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:51:45.018344 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:51:45.018880 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:51:45.018896 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:51:45.019879 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:51:45.019883 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:51:45.021211 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:51:45.021224 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:51:45.022050 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:51:45.022057 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:51:45.022266 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:45.022929 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:51:45.022962 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:51:45.025259 (+ 2297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:51:45.027997 (+ 2738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:51:45.028006 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:51:45.028021 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:51:45.028476 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:51:45.028817 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:51:45.028822 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:51:45.028825 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:51:45.028991 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:51:45.029453 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:51:45.029460 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:51:45.029760 (+ 300us) client_negotiation.cc:769] Sending connection context
0619 18:51:45.029983 (+ 223us) client_negotiation.cc:240] Negotiation successful
0619 18:51:45.030459 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":247,"threads_started":1}
I20260619 18:51:45.032910 22617 heartbeater.cc:347] Connected to a master server at 127.3.243.126:36197
I20260619 18:51:45.033203 22617 heartbeater.cc:464] Registering TS with master...
I20260619 18:51:45.034029 22617 heartbeater.cc:511] Master 127.3.243.126:36197 requested a full tablet report, sending...
I20260619 18:51:45.035754 22294 ts_manager.cc:194] Registered new tserver with Master: 70d7d7741d814dddbac57668dbc8b6e1 (127.3.243.66:44645)
I20260619 18:51:45.036667 22294 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.66@KRBTEST.COM'} at 127.3.243.66:49421
WARNING: no policy specified for kudu/127.3.243.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.67.keytab.
Entry for principal kudu/127.3.243.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.67.keytab.
I20260619 18:51:45.069484 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894810715282-4045-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894810715282-4045-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894810715282-4045-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894810715282-4045-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.67:0
--local_ip_for_outbound_sockets=127.3.243.67
--webserver_interface=127.3.243.67
--webserver_port=0
--tserver_master_addrs=127.3.243.126:36197
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.67.keytab
--principal=kudu/127.3.243.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:36469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:51:45.374758 22622 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:51:45.375001 22622 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:51:45.375102 22622 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:51:45.381920 22622 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:51:45.382007 22622 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:51:45.382118 22622 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.67
I20260619 18:51:45.391098 22622 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:36469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894810715282-4045-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894810715282-4045-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.67.keytab
--principal=kudu/127.3.243.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894810715282-4045-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.243.67
--webserver_port=0
--tserver_master_addrs=127.3.243.126:36197
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.67
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894810715282-4045-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:51:45.392189 22622 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:51:45.393433 22622 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:51:45.405411 22627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:51:45.405503 22630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:51:45.405471 22628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:51:45.406236 22622 server_base.cc:1034] running on GCE node
I20260619 18:51:45.406752 22622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:51:45.407593 22622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:51:45.408790 22622 hybrid_clock.cc:648] HybridClock initialized: now 1781895105408744 us; error 30 us; skew 500 ppm
I20260619 18:51:45.409112 22622 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:51:45 dist-test-slave-m9wg krb5kdc[22249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895105, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:51:45.411924 22622 init.cc:377] Logged in from keytab as kudu/127.3.243.67@KRBTEST.COM (short username kudu)
I20260619 18:51:45.413231 22622 webserver.cc:460] Webserver started at http://127.3.243.67:41433/ using document root <none> and password file <none>
I20260619 18:51:45.413828 22622 fs_manager.cc:359] Metadata directory not provided
I20260619 18:51:45.413935 22622 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:51:45.414242 22622 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:51:45.416939 22622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894810715282-4045-0/minicluster-data/ts-2/data/instance:
uuid: "22396c93fd3d4b51a3644e3beabe291a"
format_stamp: "Formatted at 2026-06-19 18:51:45 on dist-test-slave-m9wg"
server_key: "beefda1bd596b30a48a2816056e2efdd"
server_key_iv: "24b4ca7c1ca9cceb569296de5bdd77c6"
server_key_version: "encryptionkey@0"
I20260619 18:51:45.417635 22622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894810715282-4045-0/minicluster-data/ts-2/wal/instance:
uuid: "22396c93fd3d4b51a3644e3beabe291a"
format_stamp: "Formatted at 2026-06-19 18:51:45 on dist-test-slave-m9wg"
server_key: "beefda1bd596b30a48a2816056e2efdd"
server_key_iv: "24b4ca7c1ca9cceb569296de5bdd77c6"
server_key_version: "encryptionkey@0"
I20260619 18:51:45.422201 22622 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260619 18:51:45.425257 22637 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:51:45.426419 22622 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 18:51:45.426548 22622 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894810715282-4045-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894810715282-4045-0/minicluster-data/ts-2/wal
uuid: "22396c93fd3d4b51a3644e3beabe291a"
format_stamp: "Formatted at 2026-06-19 18:51:45 on dist-test-slave-m9wg"
server_key: "beefda1bd596b30a48a2816056e2efdd"
server_key_iv: "24b4ca7c1ca9cceb569296de5bdd77c6"
server_key_version: "encryptionkey@0"
I20260619 18:51:45.426700 22622 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894810715282-4045-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894810715282-4045-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894810715282-4045-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:51:45.447808 22622 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:51:45.451612 22622 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:51:45.451829 22622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:51:45.452658 22622 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:51:45.453830 22622 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:51:45.453899 22622 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:51:45.453986 22622 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:51:45.454028 22622 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:51:45.480731 22622 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.67:33931
I20260619 18:51:45.480772 22749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.67:33931 every 8 connection(s)
I20260619 18:51:45.481884 22622 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894810715282-4045-0/minicluster-data/ts-2/data/info.pb
I20260619 18:51:45.489077 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 22622
I20260619 18:51:45.489234 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894810715282-4045-0/minicluster-data/ts-2/wal/instance
I20260619 18:51:45.489634 4045 external_mini_cluster.cc:1428] Setting key 94c5f031ffbc99206288ab4a7cc8c5f7
Jun 19 18:51:45 dist-test-slave-m9wg krb5kdc[22249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895105, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.67@KRBTEST.COM for kudu/127.3.243.126@KRBTEST.COM
I20260619 18:51:45.499725 22485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:45.484107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.67:33741 (local address 127.3.243.126:36197)
0619 18:51:45.484336 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0619 18:51:45.484342 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:51:45.485423 (+ 1081us) server_negotiation.cc:407] Connection header received
0619 18:51:45.487103 (+ 1680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:51:45.487107 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:51:45.487189 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:51:45.487274 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:51:45.488640 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:45.489421 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:45.491106 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:45.491519 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:45.491641 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:45.494978 (+ 3337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:51:45.494983 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:51:45.494986 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:51:45.495016 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:51:45.497404 (+ 2388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:51:45.498024 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:51:45.498028 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:51:45.498030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:51:45.498139 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:51:45.498459 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:51:45.498461 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:51:45.498463 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:51:45.498696 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:51:45.498799 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 18:51:45.499350 (+ 551us) server_negotiation.cc:299] Negotiation successful
0619 18:51:45.499496 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260619 18:51:45.500614 22752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:45.484626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:36197 (local address 127.3.243.67:33741)
0619 18:51:45.485261 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:45.485301 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 18:51:45.486869 (+ 1568us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:51:45.487452 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:51:45.487460 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:51:45.487923 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:51:45.488495 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:51:45.488516 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:51:45.489561 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:51:45.489565 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:51:45.490970 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:51:45.490982 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:51:45.491725 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:51:45.491733 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:51:45.492005 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:45.492663 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:51:45.492692 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:51:45.494777 (+ 2085us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:51:45.497510 (+ 2733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:51:45.497516 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:51:45.497527 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:51:45.497936 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:51:45.498215 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:51:45.498218 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:51:45.498221 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:51:45.498381 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:51:45.498790 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:51:45.498797 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:51:45.499069 (+ 272us) client_negotiation.cc:769] Sending connection context
0619 18:51:45.499274 (+ 205us) client_negotiation.cc:240] Negotiation successful
0619 18:51:45.499614 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":234,"threads_started":1}
I20260619 18:51:45.501717 22750 heartbeater.cc:347] Connected to a master server at 127.3.243.126:36197
I20260619 18:51:45.501956 22750 heartbeater.cc:464] Registering TS with master...
I20260619 18:51:45.502658 22750 heartbeater.cc:511] Master 127.3.243.126:36197 requested a full tablet report, sending...
I20260619 18:51:45.504190 22293 ts_manager.cc:194] Registered new tserver with Master: 22396c93fd3d4b51a3644e3beabe291a (127.3.243.67:33931)
I20260619 18:51:45.505018 22293 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.67@KRBTEST.COM'} at 127.3.243.67:33741
I20260619 18:51:45.516054 4045 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 18:51:45.531277 22485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:45.519906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38228 (local address 127.3.243.126:36197)
0619 18:51:45.520052 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 18:51:45.520056 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:51:45.520463 (+ 407us) server_negotiation.cc:407] Connection header received
0619 18:51:45.520560 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:51:45.520564 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:51:45.520625 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:51:45.520707 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:51:45.521944 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:45.522624 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:45.523797 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:45.524260 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:45.524351 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:45.526664 (+ 2313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:51:45.526669 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:51:45.526672 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:51:45.526698 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:51:45.528964 (+ 2266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:51:45.529650 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:51:45.529653 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:51:45.529655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:51:45.529782 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:51:45.530131 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:51:45.530134 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:51:45.530136 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:51:45.530443 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:51:45.530631 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0619 18:51:45.530929 (+ 298us) server_negotiation.cc:299] Negotiation successful
0619 18:51:45.531050 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 18:51:45.534731 22293 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38228:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:51:45.541276 22293 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:51:45.565725 22483 heartbeater.cc:503] Master 127.3.243.126:36197 was elected leader, sending a full tablet report...
I20260619 18:51:45.574977 22763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:45.552261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40366 (local address 127.3.243.65:35817)
0619 18:51:45.564847 (+ 12586us) server_negotiation.cc:206] Beginning negotiation
0619 18:51:45.564857 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:51:45.566173 (+ 1316us) server_negotiation.cc:407] Connection header received
0619 18:51:45.566607 (+ 434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:51:45.566630 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:51:45.566910 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:51:45.567074 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:51:45.568116 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:45.569719 (+ 1603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:45.572380 (+ 2661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:45.573811 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:45.573941 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:45.574183 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0619 18:51:45.574486 (+ 303us) server_negotiation.cc:299] Negotiation successful
0619 18:51:45.574663 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12385,"thread_start_us":12612,"threads_started":1}
I20260619 18:51:45.575054 22762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:45.552092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.65:35817 (local address 127.0.0.1:40366)
0619 18:51:45.566013 (+ 13921us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:45.566062 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 18:51:45.566347 (+ 285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:51:45.567400 (+ 1053us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:51:45.567408 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:51:45.567442 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:51:45.567938 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:51:45.567953 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:51:45.569875 (+ 1922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:51:45.569881 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:51:45.572215 (+ 2334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:51:45.572230 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:51:45.574021 (+ 1791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:51:45.574027 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:51:45.574233 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:45.574288 (+ 55us) client_negotiation.cc:769] Sending connection context
0619 18:51:45.574414 (+ 126us) client_negotiation.cc:240] Negotiation successful
0619 18:51:45.574595 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13679,"mutex_wait_us":7,"thread_start_us":13544,"threads_started":1}
I20260619 18:51:45.577481 22766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:45.552824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53088 (local address 127.3.243.66:44645)
0619 18:51:45.565498 (+ 12674us) server_negotiation.cc:206] Beginning negotiation
0619 18:51:45.565506 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:51:45.569217 (+ 3711us) server_negotiation.cc:407] Connection header received
0619 18:51:45.569388 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:51:45.569395 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:51:45.569625 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:51:45.569815 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:51:45.570831 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:45.571984 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:45.574682 (+ 2698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:45.576049 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:45.576194 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:45.576382 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0619 18:51:45.576953 (+ 571us) server_negotiation.cc:299] Negotiation successful
0619 18:51:45.577174 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12497,"thread_start_us":12368,"threads_started":1}
I20260619 18:51:45.577590 22767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:45.553370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51966 (local address 127.3.243.67:33931)
0619 18:51:45.568641 (+ 15271us) server_negotiation.cc:206] Beginning negotiation
0619 18:51:45.568651 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:51:45.569916 (+ 1265us) server_negotiation.cc:407] Connection header received
0619 18:51:45.570012 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:51:45.570018 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:51:45.570239 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:51:45.570407 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:51:45.571320 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:45.572767 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:45.575047 (+ 2280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:45.576046 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:45.576183 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:45.576417 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0619 18:51:45.577212 (+ 795us) server_negotiation.cc:299] Negotiation successful
0619 18:51:45.577334 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15073,"thread_start_us":14902,"threads_started":1}
I20260619 18:51:45.578291 22764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:45.552758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.67:33931 (local address 127.0.0.1:51966)
0619 18:51:45.569677 (+ 16919us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:45.569692 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 18:51:45.569803 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:51:45.570588 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:51:45.570592 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:51:45.570623 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:51:45.571131 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:51:45.571144 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:51:45.572920 (+ 1776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:51:45.572926 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:51:45.574855 (+ 1929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:51:45.574869 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:51:45.576729 (+ 1860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:51:45.576735 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:51:45.576907 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:45.576937 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 18:51:45.577018 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 18:51:45.577916 (+ 898us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16802,"mutex_wait_us":29,"thread_start_us":16037,"threads_started":1}
I20260619 18:51:45.578329 22765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:45.552301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.66:44645 (local address 127.0.0.1:53088)
0619 18:51:45.569087 (+ 16786us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:45.569106 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 18:51:45.569234 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:51:45.570070 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:51:45.570075 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:51:45.570099 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:51:45.570560 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:51:45.570576 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:51:45.572125 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:51:45.572129 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:51:45.574516 (+ 2387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:51:45.574529 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:51:45.576565 (+ 2036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:51:45.576571 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:51:45.576754 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:45.576786 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 18:51:45.577719 (+ 933us) client_negotiation.cc:240] Negotiation successful
0619 18:51:45.577841 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16644,"mutex_wait_us":34,"thread_start_us":17140,"threads_started":1}
I20260619 18:51:45.579253 22418 tablet_service.cc:1467] Processing CreateTablet for tablet 41ec1eddb6f8487393b6f3d6082a75b4 (DEFAULT_TABLE table=test-table [id=6d156b593970479da92988dcc49ce6df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:51:45.580641 22418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41ec1eddb6f8487393b6f3d6082a75b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:51:45.581815 22552 tablet_service.cc:1467] Processing CreateTablet for tablet 41ec1eddb6f8487393b6f3d6082a75b4 (DEFAULT_TABLE table=test-table [id=6d156b593970479da92988dcc49ce6df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:51:45.581815 22685 tablet_service.cc:1467] Processing CreateTablet for tablet 41ec1eddb6f8487393b6f3d6082a75b4 (DEFAULT_TABLE table=test-table [id=6d156b593970479da92988dcc49ce6df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:51:45.583120 22552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41ec1eddb6f8487393b6f3d6082a75b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:51:45.583119 22685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41ec1eddb6f8487393b6f3d6082a75b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:51:45.592487 22768 tablet_bootstrap.cc:492] T 41ec1eddb6f8487393b6f3d6082a75b4 P fb235187b06a41c7aa323544a09a5a10: Bootstrap starting.
I20260619 18:51:45.595405 22768 tablet_bootstrap.cc:654] T 41ec1eddb6f8487393b6f3d6082a75b4 P fb235187b06a41c7aa323544a09a5a10: Neither blocks nor log segments found. Creating new log.
I20260619 18:51:45.596465 22768 log.cc:824] T 41ec1eddb6f8487393b6f3d6082a75b4 P fb235187b06a41c7aa323544a09a5a10: Log is configured to *not* fsync() on all Append() calls
I20260619 18:51:45.597461 22769 tablet_bootstrap.cc:492] T 41ec1eddb6f8487393b6f3d6082a75b4 P 70d7d7741d814dddbac57668dbc8b6e1: Bootstrap starting.
I20260619 18:51:45.601191 22769 tablet_bootstrap.cc:654] T 41ec1eddb6f8487393b6f3d6082a75b4 P 70d7d7741d814dddbac57668dbc8b6e1: Neither blocks nor log segments found. Creating new log.
I20260619 18:51:45.602464 22769 log.cc:824] T 41ec1eddb6f8487393b6f3d6082a75b4 P 70d7d7741d814dddbac57668dbc8b6e1: Log is configured to *not* fsync() on all Append() calls
I20260619 18:51:45.602843 22770 tablet_bootstrap.cc:492] T 41ec1eddb6f8487393b6f3d6082a75b4 P 22396c93fd3d4b51a3644e3beabe291a: Bootstrap starting.
I20260619 18:51:45.605692 22770 tablet_bootstrap.cc:654] T 41ec1eddb6f8487393b6f3d6082a75b4 P 22396c93fd3d4b51a3644e3beabe291a: Neither blocks nor log segments found. Creating new log.
I20260619 18:51:45.606444 22768 tablet_bootstrap.cc:492] T 41ec1eddb6f8487393b6f3d6082a75b4 P fb235187b06a41c7aa323544a09a5a10: No bootstrap required, opened a new log
I20260619 18:51:45.606710 22768 ts_tablet_manager.cc:1397] T 41ec1eddb6f8487393b6f3d6082a75b4 P fb235187b06a41c7aa323544a09a5a10: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.006s
I20260619 18:51:45.606901 22769 tablet_bootstrap.cc:492] T 41ec1eddb6f8487393b6f3d6082a75b4 P 70d7d7741d814dddbac57668dbc8b6e1: No bootstrap required, opened a new log
I20260619 18:51:45.606839 22770 log.cc:824] T 41ec1eddb6f8487393b6f3d6082a75b4 P 22396c93fd3d4b51a3644e3beabe291a: Log is configured to *not* fsync() on all Append() calls
I20260619 18:51:45.607133 22769 ts_tablet_manager.cc:1397] T 41ec1eddb6f8487393b6f3d6082a75b4 P 70d7d7741d814dddbac57668dbc8b6e1: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.002s
I20260619 18:51:45.610394 22770 tablet_bootstrap.cc:492] T 41ec1eddb6f8487393b6f3d6082a75b4 P 22396c93fd3d4b51a3644e3beabe291a: No bootstrap required, opened a new log
I20260619 18:51:45.610642 22770 ts_tablet_manager.cc:1397] T 41ec1eddb6f8487393b6f3d6082a75b4 P 22396c93fd3d4b51a3644e3beabe291a: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260619 18:51:45.614295 22768 raft_consensus.cc:348] T 41ec1eddb6f8487393b6f3d6082a75b4 P fb235187b06a41c7aa323544a09a5a10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb235187b06a41c7aa323544a09a5a10" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 35817 } } peers { permanent_uuid: "70d7d7741d814dddbac57668dbc8b6e1" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 44645 } } peers { permanent_uuid: "22396c93fd3d4b51a3644e3beabe291a" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 33931 } }
I20260619 18:51:45.614648 22768 raft_consensus.cc:374] T 41ec1eddb6f8487393b6f3d6082a75b4 P fb235187b06a41c7aa323544a09a5a10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:51:45.614773 22768 raft_consensus.cc:729] T 41ec1eddb6f8487393b6f3d6082a75b4 P fb235187b06a41c7aa323544a09a5a10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb235187b06a41c7aa323544a09a5a10, State: Initialized, Role: FOLLOWER
I20260619 18:51:45.615320 22768 consensus_queue.cc:260] T 41ec1eddb6f8487393b6f3d6082a75b4 P fb235187b06a41c7aa323544a09a5a10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb235187b06a41c7aa323544a09a5a10" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 35817 } } peers { permanent_uuid: "70d7d7741d814dddbac57668dbc8b6e1" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 44645 } } peers { permanent_uuid: "22396c93fd3d4b51a3644e3beabe291a" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 33931 } }
I20260619 18:51:45.616990 22768 ts_tablet_manager.cc:1428] T 41ec1eddb6f8487393b6f3d6082a75b4 P fb235187b06a41c7aa323544a09a5a10: Time spent starting tablet: real 0.010s user 0.009s sys 0.001s
I20260619 18:51:45.616959 22769 raft_consensus.cc:348] T 41ec1eddb6f8487393b6f3d6082a75b4 P 70d7d7741d814dddbac57668dbc8b6e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb235187b06a41c7aa323544a09a5a10" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 35817 } } peers { permanent_uuid: "70d7d7741d814dddbac57668dbc8b6e1" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 44645 } } peers { permanent_uuid: "22396c93fd3d4b51a3644e3beabe291a" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 33931 } }
I20260619 18:51:45.617494 22769 raft_consensus.cc:374] T 41ec1eddb6f8487393b6f3d6082a75b4 P 70d7d7741d814dddbac57668dbc8b6e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:51:45.617623 22769 raft_consensus.cc:729] T 41ec1eddb6f8487393b6f3d6082a75b4 P 70d7d7741d814dddbac57668dbc8b6e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70d7d7741d814dddbac57668dbc8b6e1, State: Initialized, Role: FOLLOWER
I20260619 18:51:45.618436 22769 consensus_queue.cc:260] T 41ec1eddb6f8487393b6f3d6082a75b4 P 70d7d7741d814dddbac57668dbc8b6e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb235187b06a41c7aa323544a09a5a10" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 35817 } } peers { permanent_uuid: "70d7d7741d814dddbac57668dbc8b6e1" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 44645 } } peers { permanent_uuid: "22396c93fd3d4b51a3644e3beabe291a" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 33931 } }
I20260619 18:51:45.619513 22617 heartbeater.cc:503] Master 127.3.243.126:36197 was elected leader, sending a full tablet report...
I20260619 18:51:45.620136 22769 ts_tablet_manager.cc:1428] T 41ec1eddb6f8487393b6f3d6082a75b4 P 70d7d7741d814dddbac57668dbc8b6e1: Time spent starting tablet: real 0.013s user 0.007s sys 0.006s
I20260619 18:51:45.620600 22770 raft_consensus.cc:348] T 41ec1eddb6f8487393b6f3d6082a75b4 P 22396c93fd3d4b51a3644e3beabe291a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb235187b06a41c7aa323544a09a5a10" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 35817 } } peers { permanent_uuid: "70d7d7741d814dddbac57668dbc8b6e1" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 44645 } } peers { permanent_uuid: "22396c93fd3d4b51a3644e3beabe291a" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 33931 } }
I20260619 18:51:45.620995 22770 raft_consensus.cc:374] T 41ec1eddb6f8487393b6f3d6082a75b4 P 22396c93fd3d4b51a3644e3beabe291a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:51:45.621086 22770 raft_consensus.cc:729] T 41ec1eddb6f8487393b6f3d6082a75b4 P 22396c93fd3d4b51a3644e3beabe291a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22396c93fd3d4b51a3644e3beabe291a, State: Initialized, Role: FOLLOWER
I20260619 18:51:45.621649 22770 consensus_queue.cc:260] T 41ec1eddb6f8487393b6f3d6082a75b4 P 22396c93fd3d4b51a3644e3beabe291a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb235187b06a41c7aa323544a09a5a10" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 35817 } } peers { permanent_uuid: "70d7d7741d814dddbac57668dbc8b6e1" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 44645 } } peers { permanent_uuid: "22396c93fd3d4b51a3644e3beabe291a" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 33931 } }
I20260619 18:51:45.622714 22750 heartbeater.cc:503] Master 127.3.243.126:36197 was elected leader, sending a full tablet report...
I20260619 18:51:45.623260 22770 ts_tablet_manager.cc:1428] T 41ec1eddb6f8487393b6f3d6082a75b4 P 22396c93fd3d4b51a3644e3beabe291a: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
W20260619 18:51:45.733333 22751 tablet.cc:2369] T 41ec1eddb6f8487393b6f3d6082a75b4 P 22396c93fd3d4b51a3644e3beabe291a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 18:51:45.763618 22618 tablet.cc:2369] T 41ec1eddb6f8487393b6f3d6082a75b4 P 70d7d7741d814dddbac57668dbc8b6e1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 18:51:45.789084 22484 tablet.cc:2369] T 41ec1eddb6f8487393b6f3d6082a75b4 P fb235187b06a41c7aa323544a09a5a10: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:51:45.870656 22776 raft_consensus.cc:482] T 41ec1eddb6f8487393b6f3d6082a75b4 P 22396c93fd3d4b51a3644e3beabe291a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:51:45.870945 22776 raft_consensus.cc:504] T 41ec1eddb6f8487393b6f3d6082a75b4 P 22396c93fd3d4b51a3644e3beabe291a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb235187b06a41c7aa323544a09a5a10" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 35817 } } peers { permanent_uuid: "70d7d7741d814dddbac57668dbc8b6e1" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 44645 } } peers { permanent_uuid: "22396c93fd3d4b51a3644e3beabe291a" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 33931 } }
I20260619 18:51:45.872596 22776 leader_election.cc:288] T 41ec1eddb6f8487393b6f3d6082a75b4 P 22396c93fd3d4b51a3644e3beabe291a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fb235187b06a41c7aa323544a09a5a10 (127.3.243.65:35817), 70d7d7741d814dddbac57668dbc8b6e1 (127.3.243.66:44645)
I20260619 18:51:45.877807 22752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:45.872903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.66:44645 (local address 127.3.243.67:34987)
0619 18:51:45.873234 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:45.873256 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 18:51:45.873388 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:51:45.873701 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:51:45.873705 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:51:45.873731 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:51:45.873895 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:51:45.873903 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:51:45.874846 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:51:45.874852 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:51:45.876115 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:51:45.876126 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:51:45.877138 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:51:45.877144 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:51:45.877303 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:45.877334 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 18:51:45.877409 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 18:51:45.877528 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199}
I20260619 18:51:45.877792 22766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:45.873078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.67:34987 (local address 127.3.243.66:44645)
0619 18:51:45.873259 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 18:51:45.873264 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:51:45.873308 (+ 44us) server_negotiation.cc:407] Connection header received
0619 18:51:45.873532 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:51:45.873536 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:51:45.873594 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:51:45.873666 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:51:45.874026 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:45.874721 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:45.876279 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:45.877003 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:45.877135 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:45.877205 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 18:51:45.877447 (+ 242us) server_negotiation.cc:299] Negotiation successful
0619 18:51:45.877565 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 18:51:45.878456 22763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:45.872971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.67:59197 (local address 127.3.243.65:35817)
0619 18:51:45.873181 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0619 18:51:45.873187 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:51:45.873754 (+ 567us) server_negotiation.cc:407] Connection header received
0619 18:51:45.873925 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:51:45.873929 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:51:45.873986 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:51:45.874059 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:51:45.874556 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:45.875224 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:45.876880 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:45.877629 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:45.877795 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:45.877863 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 18:51:45.878145 (+ 282us) server_negotiation.cc:299] Negotiation successful
0619 18:51:45.878271 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 18:51:45.878557 22777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:45.872948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.65:35817 (local address 127.3.243.67:59197)
0619 18:51:45.873658 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:45.873671 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 18:51:45.873803 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:51:45.874080 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:51:45.874084 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:51:45.874104 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:51:45.874418 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:51:45.874426 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:51:45.875371 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:51:45.875374 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:51:45.876738 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:51:45.876750 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:51:45.877818 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:51:45.877823 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:51:45.877981 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:45.878011 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 18:51:45.878104 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 18:51:45.878217 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":622,"mutex_wait_us":1,"thread_start_us":392,"threads_started":1}
I20260619 18:51:45.878950 22572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41ec1eddb6f8487393b6f3d6082a75b4" candidate_uuid: "22396c93fd3d4b51a3644e3beabe291a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70d7d7741d814dddbac57668dbc8b6e1" is_pre_election: true
I20260619 18:51:45.879416 22572 raft_consensus.cc:2454] T 41ec1eddb6f8487393b6f3d6082a75b4 P 70d7d7741d814dddbac57668dbc8b6e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22396c93fd3d4b51a3644e3beabe291a in term 0.
I20260619 18:51:45.879434 22438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41ec1eddb6f8487393b6f3d6082a75b4" candidate_uuid: "22396c93fd3d4b51a3644e3beabe291a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb235187b06a41c7aa323544a09a5a10" is_pre_election: true
I20260619 18:51:45.879933 22438 raft_consensus.cc:2454] T 41ec1eddb6f8487393b6f3d6082a75b4 P fb235187b06a41c7aa323544a09a5a10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22396c93fd3d4b51a3644e3beabe291a in term 0.
I20260619 18:51:45.879998 22639 leader_election.cc:302] T 41ec1eddb6f8487393b6f3d6082a75b4 P 22396c93fd3d4b51a3644e3beabe291a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22396c93fd3d4b51a3644e3beabe291a, 70d7d7741d814dddbac57668dbc8b6e1; no voters:
I20260619 18:51:45.880362 22776 raft_consensus.cc:2781] T 41ec1eddb6f8487393b6f3d6082a75b4 P 22396c93fd3d4b51a3644e3beabe291a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:51:45.880481 22776 raft_consensus.cc:482] T 41ec1eddb6f8487393b6f3d6082a75b4 P 22396c93fd3d4b51a3644e3beabe291a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:51:45.880579 22776 raft_consensus.cc:3037] T 41ec1eddb6f8487393b6f3d6082a75b4 P 22396c93fd3d4b51a3644e3beabe291a [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:51:45.882647 22776 raft_consensus.cc:504] T 41ec1eddb6f8487393b6f3d6082a75b4 P 22396c93fd3d4b51a3644e3beabe291a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb235187b06a41c7aa323544a09a5a10" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 35817 } } peers { permanent_uuid: "70d7d7741d814dddbac57668dbc8b6e1" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 44645 } } peers { permanent_uuid: "22396c93fd3d4b51a3644e3beabe291a" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 33931 } }
I20260619 18:51:45.883176 22776 leader_election.cc:288] T 41ec1eddb6f8487393b6f3d6082a75b4 P 22396c93fd3d4b51a3644e3beabe291a [CANDIDATE]: Term 1 election: Requested vote from peers fb235187b06a41c7aa323544a09a5a10 (127.3.243.65:35817), 70d7d7741d814dddbac57668dbc8b6e1 (127.3.243.66:44645)
I20260619 18:51:45.883520 22438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41ec1eddb6f8487393b6f3d6082a75b4" candidate_uuid: "22396c93fd3d4b51a3644e3beabe291a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb235187b06a41c7aa323544a09a5a10"
I20260619 18:51:45.883625 22572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41ec1eddb6f8487393b6f3d6082a75b4" candidate_uuid: "22396c93fd3d4b51a3644e3beabe291a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70d7d7741d814dddbac57668dbc8b6e1"
I20260619 18:51:45.883698 22438 raft_consensus.cc:3037] T 41ec1eddb6f8487393b6f3d6082a75b4 P fb235187b06a41c7aa323544a09a5a10 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:51:45.883808 22572 raft_consensus.cc:3037] T 41ec1eddb6f8487393b6f3d6082a75b4 P 70d7d7741d814dddbac57668dbc8b6e1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:51:45.885838 22438 raft_consensus.cc:2454] T 41ec1eddb6f8487393b6f3d6082a75b4 P fb235187b06a41c7aa323544a09a5a10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22396c93fd3d4b51a3644e3beabe291a in term 1.
I20260619 18:51:45.885848 22572 raft_consensus.cc:2454] T 41ec1eddb6f8487393b6f3d6082a75b4 P 70d7d7741d814dddbac57668dbc8b6e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22396c93fd3d4b51a3644e3beabe291a in term 1.
I20260619 18:51:45.886200 22638 leader_election.cc:302] T 41ec1eddb6f8487393b6f3d6082a75b4 P 22396c93fd3d4b51a3644e3beabe291a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 22396c93fd3d4b51a3644e3beabe291a, 70d7d7741d814dddbac57668dbc8b6e1, fb235187b06a41c7aa323544a09a5a10; no voters:
I20260619 18:51:45.886503 22776 raft_consensus.cc:2781] T 41ec1eddb6f8487393b6f3d6082a75b4 P 22396c93fd3d4b51a3644e3beabe291a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:51:45.886900 22776 raft_consensus.cc:686] T 41ec1eddb6f8487393b6f3d6082a75b4 P 22396c93fd3d4b51a3644e3beabe291a [term 1 LEADER]: Becoming Leader. State: Replica: 22396c93fd3d4b51a3644e3beabe291a, State: Running, Role: LEADER
I20260619 18:51:45.887320 22776 consensus_queue.cc:237] T 41ec1eddb6f8487393b6f3d6082a75b4 P 22396c93fd3d4b51a3644e3beabe291a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb235187b06a41c7aa323544a09a5a10" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 35817 } } peers { permanent_uuid: "70d7d7741d814dddbac57668dbc8b6e1" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 44645 } } peers { permanent_uuid: "22396c93fd3d4b51a3644e3beabe291a" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 33931 } }
I20260619 18:51:45.891414 22294 catalog_manager.cc:5495] T 41ec1eddb6f8487393b6f3d6082a75b4 P 22396c93fd3d4b51a3644e3beabe291a reported cstate change: term changed from 0 to 1, leader changed from <none> to 22396c93fd3d4b51a3644e3beabe291a (127.3.243.67). New cstate: current_term: 1 leader_uuid: "22396c93fd3d4b51a3644e3beabe291a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb235187b06a41c7aa323544a09a5a10" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 35817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "70d7d7741d814dddbac57668dbc8b6e1" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 44645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22396c93fd3d4b51a3644e3beabe291a" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 33931 } health_report { overall_health: HEALTHY } } }
I20260619 18:51:45.942662 22767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:45.937437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51974 (local address 127.3.243.67:33931)
0619 18:51:45.937618 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 18:51:45.937623 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:51:45.937642 (+ 19us) server_negotiation.cc:407] Connection header received
0619 18:51:45.937779 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:51:45.937785 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:51:45.937882 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:51:45.938006 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:51:45.938375 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:45.939072 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:45.940497 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:45.940905 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:45.941011 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:45.941585 (+ 574us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:51:45.942063 (+ 478us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:51:45.942218 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0619 18:51:45.942376 (+ 158us) server_negotiation.cc:299] Negotiation successful
0619 18:51:45.942423 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 18:51:45.950007 22572 raft_consensus.cc:1261] T 41ec1eddb6f8487393b6f3d6082a75b4 P 70d7d7741d814dddbac57668dbc8b6e1 [term 1 FOLLOWER]: Refusing update from remote peer 22396c93fd3d4b51a3644e3beabe291a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:51:45.950052 22438 raft_consensus.cc:1261] T 41ec1eddb6f8487393b6f3d6082a75b4 P fb235187b06a41c7aa323544a09a5a10 [term 1 FOLLOWER]: Refusing update from remote peer 22396c93fd3d4b51a3644e3beabe291a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:51:45.950875 22778 consensus_queue.cc:1035] T 41ec1eddb6f8487393b6f3d6082a75b4 P 22396c93fd3d4b51a3644e3beabe291a [LEADER]: Connected to new peer: Peer: permanent_uuid: "70d7d7741d814dddbac57668dbc8b6e1" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:51:45.951184 22776 consensus_queue.cc:1035] T 41ec1eddb6f8487393b6f3d6082a75b4 P 22396c93fd3d4b51a3644e3beabe291a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb235187b06a41c7aa323544a09a5a10" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 35817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:51:45.960217 22781 mvcc.cc:205] Tried to move back new op lower bound from 7298642353962639360 to 7298642353717760000. Current Snapshot: MvccSnapshot[applied={T|T < 7298642353962639360}]
I20260619 18:51:45.960551 22782 mvcc.cc:205] Tried to move back new op lower bound from 7298642353962639360 to 7298642353717760000. Current Snapshot: MvccSnapshot[applied={T|T < 7298642353962639360}]
I20260619 18:51:45.975011 22294 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38228:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:51:45.975255 22294 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38228:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:51:45.978622 22294 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 00c2537b1903461d9ba9f07c5e0b9e3c: Sending DeleteTablet for 3 replicas of tablet 41ec1eddb6f8487393b6f3d6082a75b4
I20260619 18:51:45.979756 22552 tablet_service.cc:1514] Processing DeleteTablet for tablet 41ec1eddb6f8487393b6f3d6082a75b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:51:45 UTC) from {username='kudu', principal='kudu/127.3.243.126@KRBTEST.COM'} at 127.0.0.1:53088
I20260619 18:51:45.980075 22418 tablet_service.cc:1514] Processing DeleteTablet for tablet 41ec1eddb6f8487393b6f3d6082a75b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:51:45 UTC) from {username='kudu', principal='kudu/127.3.243.126@KRBTEST.COM'} at 127.0.0.1:40366
I20260619 18:51:45.981632 22685 tablet_service.cc:1514] Processing DeleteTablet for tablet 41ec1eddb6f8487393b6f3d6082a75b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:51:45 UTC) from {username='kudu', principal='kudu/127.3.243.126@KRBTEST.COM'} at 127.0.0.1:51966
I20260619 18:51:45.988756 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 22355
I20260619 18:51:46.002014 22792 tablet_replica.cc:330] T 41ec1eddb6f8487393b6f3d6082a75b4 P 22396c93fd3d4b51a3644e3beabe291a: stopping tablet replica
I20260619 18:51:46.002478 22792 raft_consensus.cc:2229] T 41ec1eddb6f8487393b6f3d6082a75b4 P 22396c93fd3d4b51a3644e3beabe291a [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:51:46.003151 22792 raft_consensus.cc:2258] T 41ec1eddb6f8487393b6f3d6082a75b4 P 22396c93fd3d4b51a3644e3beabe291a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:51:46.003851 22791 tablet_replica.cc:330] T 41ec1eddb6f8487393b6f3d6082a75b4 P fb235187b06a41c7aa323544a09a5a10: stopping tablet replica
I20260619 18:51:46.004297 22791 raft_consensus.cc:2229] T 41ec1eddb6f8487393b6f3d6082a75b4 P fb235187b06a41c7aa323544a09a5a10 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:51:46.004809 22791 raft_consensus.cc:2258] T 41ec1eddb6f8487393b6f3d6082a75b4 P fb235187b06a41c7aa323544a09a5a10 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:51:46.006721 22792 ts_tablet_manager.cc:1905] T 41ec1eddb6f8487393b6f3d6082a75b4 P 22396c93fd3d4b51a3644e3beabe291a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:51:46.009717 22791 ts_tablet_manager.cc:1905] T 41ec1eddb6f8487393b6f3d6082a75b4 P fb235187b06a41c7aa323544a09a5a10: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:51:46.018314 22792 ts_tablet_manager.cc:1918] T 41ec1eddb6f8487393b6f3d6082a75b4 P 22396c93fd3d4b51a3644e3beabe291a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:51:46.022874 22791 ts_tablet_manager.cc:1918] T 41ec1eddb6f8487393b6f3d6082a75b4 P fb235187b06a41c7aa323544a09a5a10: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:51:46.023038 22791 log.cc:1196] T 41ec1eddb6f8487393b6f3d6082a75b4 P fb235187b06a41c7aa323544a09a5a10: Deleting WAL directory at /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894810715282-4045-0/minicluster-data/ts-0/wal/wals/41ec1eddb6f8487393b6f3d6082a75b4
I20260619 18:51:46.023553 22791 ts_tablet_manager.cc:1939] T 41ec1eddb6f8487393b6f3d6082a75b4 P fb235187b06a41c7aa323544a09a5a10: Deleting consensus metadata
I20260619 18:51:46.025468 22278 catalog_manager.cc:4841] TS fb235187b06a41c7aa323544a09a5a10 (127.3.243.65:35817): tablet 41ec1eddb6f8487393b6f3d6082a75b4 (table test-table [id=6d156b593970479da92988dcc49ce6df]) successfully deleted
I20260619 18:51:46.026196 22790 tablet_replica.cc:330] T 41ec1eddb6f8487393b6f3d6082a75b4 P 70d7d7741d814dddbac57668dbc8b6e1: stopping tablet replica
I20260619 18:51:46.026615 22790 raft_consensus.cc:2229] T 41ec1eddb6f8487393b6f3d6082a75b4 P 70d7d7741d814dddbac57668dbc8b6e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:51:46.027112 22790 raft_consensus.cc:2258] T 41ec1eddb6f8487393b6f3d6082a75b4 P 70d7d7741d814dddbac57668dbc8b6e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:51:46.027595 22792 log.cc:1196] T 41ec1eddb6f8487393b6f3d6082a75b4 P 22396c93fd3d4b51a3644e3beabe291a: Deleting WAL directory at /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894810715282-4045-0/minicluster-data/ts-2/wal/wals/41ec1eddb6f8487393b6f3d6082a75b4
Jun 19 18:51:46 dist-test-slave-m9wg krb5kdc[22249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895103, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.65@KRBTEST.COM
I20260619 18:51:46.028139 22792 ts_tablet_manager.cc:1939] T 41ec1eddb6f8487393b6f3d6082a75b4 P 22396c93fd3d4b51a3644e3beabe291a: Deleting consensus metadata
I20260619 18:51:46.029908 22280 catalog_manager.cc:4841] TS 22396c93fd3d4b51a3644e3beabe291a (127.3.243.67:33931): tablet 41ec1eddb6f8487393b6f3d6082a75b4 (table test-table [id=6d156b593970479da92988dcc49ce6df]) successfully deleted
I20260619 18:51:46.030941 22790 ts_tablet_manager.cc:1905] T 41ec1eddb6f8487393b6f3d6082a75b4 P 70d7d7741d814dddbac57668dbc8b6e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:51:46.036028 22763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:45.990256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40376 (local address 127.3.243.65:35817)
0619 18:51:45.990438 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 18:51:45.990445 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:51:46.001350 (+ 10905us) server_negotiation.cc:407] Connection header received
0619 18:51:46.001560 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:51:46.001566 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:51:46.001659 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:51:46.001760 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:51:46.006660 (+ 4900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:46.007843 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:46.023762 (+ 15919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:46.024499 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:46.024605 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:46.029153 (+ 4548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:51:46.029167 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:51:46.029185 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:51:46.029223 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:51:46.032784 (+ 3561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:51:46.033643 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:51:46.033652 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:51:46.033656 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:51:46.033794 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:51:46.034206 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:51:46.034211 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:51:46.034215 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:51:46.034630 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:51:46.034900 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0619 18:51:46.035471 (+ 571us) server_negotiation.cc:299] Negotiation successful
0619 18:51:46.035682 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 18:51:46.038059 22790 ts_tablet_manager.cc:1918] T 41ec1eddb6f8487393b6f3d6082a75b4 P 70d7d7741d814dddbac57668dbc8b6e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:51:46.038209 22790 log.cc:1196] T 41ec1eddb6f8487393b6f3d6082a75b4 P 70d7d7741d814dddbac57668dbc8b6e1: Deleting WAL directory at /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894810715282-4045-0/minicluster-data/ts-1/wal/wals/41ec1eddb6f8487393b6f3d6082a75b4
I20260619 18:51:46.038754 22790 ts_tablet_manager.cc:1939] T 41ec1eddb6f8487393b6f3d6082a75b4 P 70d7d7741d814dddbac57668dbc8b6e1: Deleting consensus metadata
I20260619 18:51:46.039371 22478 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:51:46.040642 22279 catalog_manager.cc:4841] TS 70d7d7741d814dddbac57668dbc8b6e1 (127.3.243.66:44645): tablet 41ec1eddb6f8487393b6f3d6082a75b4 (table test-table [id=6d156b593970479da92988dcc49ce6df]) successfully deleted
I20260619 18:51:46.126655 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 22355
I20260619 18:51:46.141263 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 22489
Jun 19 18:51:46 dist-test-slave-m9wg krb5kdc[22249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895103, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.66@KRBTEST.COM
I20260619 18:51:46.155707 22766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:46.142026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53092 (local address 127.3.243.66:44645)
0619 18:51:46.142202 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 18:51:46.142208 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:51:46.142226 (+ 18us) server_negotiation.cc:407] Connection header received
0619 18:51:46.142366 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:51:46.142372 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:51:46.142450 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:51:46.142532 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:51:46.143735 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:46.144497 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:46.145740 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:46.146140 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:46.146243 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:46.150403 (+ 4160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:51:46.150417 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:51:46.150432 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:51:46.150463 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:51:46.153156 (+ 2693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:51:46.153765 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:51:46.153772 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:51:46.153775 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:51:46.153905 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:51:46.154253 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:51:46.154257 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:51:46.154259 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:51:46.154616 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:51:46.154912 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0619 18:51:46.155169 (+ 257us) server_negotiation.cc:299] Negotiation successful
0619 18:51:46.155391 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 18:51:46.156389 22612 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:51:46.242457 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 22489
I20260619 18:51:46.256680 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 22622
Jun 19 18:51:46 dist-test-slave-m9wg krb5kdc[22249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895103, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.67@KRBTEST.COM
I20260619 18:51:46.274179 22767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:46.257511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51984 (local address 127.3.243.67:33931)
0619 18:51:46.257679 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 18:51:46.257683 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:51:46.257754 (+ 71us) server_negotiation.cc:407] Connection header received
0619 18:51:46.257990 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:51:46.257998 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:51:46.258069 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:51:46.258150 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:51:46.259939 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:46.260754 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:46.262518 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:46.262958 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:46.263043 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:46.268522 (+ 5479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:51:46.268537 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:51:46.268552 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:51:46.268587 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:51:46.271382 (+ 2795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:51:46.272256 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:51:46.272262 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:51:46.272265 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:51:46.272412 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:51:46.272828 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:51:46.272831 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:51:46.272833 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:51:46.273164 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:51:46.273404 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0619 18:51:46.273694 (+ 290us) server_negotiation.cc:299] Negotiation successful
0619 18:51:46.273853 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 18:51:46.274791 22745 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:51:46.393697 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 22622
I20260619 18:51:46.409452 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 22262
I20260619 18:51:46.410434 22324 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:51:46.517352 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 22262
2026-06-19T18:51:46Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4175 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-19T18:51:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:51:46Z Disabled control of system clock
I20260619 18:51:46.583235 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894810715282-4045-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894810715282-4045-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894810715282-4045-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.126:44063
--webserver_interface=127.3.243.126
--webserver_port=0
--builtin_ntp_servers=127.3.243.84:39311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.126:44063
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 18:51:46.888000 22801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:51:46.888273 22801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:51:46.888387 22801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:51:46.895251 22801 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:51:46.895339 22801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:51:46.895402 22801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:51:46.895473 22801 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:51:46.895534 22801 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:51:46.904247 22801 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:39311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894810715282-4045-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894810715282-4045-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.243.126:44063
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.126:44063
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.243.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894810715282-4045-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:51:46.905423 22801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:51:46.907061 22801 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:51:46.917549 22806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:51:46.917629 22807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:51:46.917676 22809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:51:46.918541 22801 server_base.cc:1034] running on GCE node
I20260619 18:51:46.919269 22801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:51:46.920593 22801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:51:46.921794 22801 hybrid_clock.cc:648] HybridClock initialized: now 1781895106921745 us; error 41 us; skew 500 ppm
I20260619 18:51:46.922102 22801 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 18:51:46.924093 22801 webserver.cc:460] Webserver started at http://127.3.243.126:35209/ using document root <none> and password file <none>
I20260619 18:51:46.924731 22801 fs_manager.cc:359] Metadata directory not provided
I20260619 18:51:46.924825 22801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:51:46.925114 22801 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:51:46.927764 22801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894810715282-4045-0/minicluster-data/master-0/data/instance:
uuid: "dc88dbc207444145a2234536d1da901c"
format_stamp: "Formatted at 2026-06-19 18:51:46 on dist-test-slave-m9wg"
server_key: "37b3e93e90d88faea0c508a2edff0577"
server_key_iv: "eb2673148625bbe79bf5b7a92bd9fab3"
server_key_version: "encryptionkey@0"
I20260619 18:51:46.928545 22801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894810715282-4045-0/minicluster-data/master-0/wal/instance:
uuid: "dc88dbc207444145a2234536d1da901c"
format_stamp: "Formatted at 2026-06-19 18:51:46 on dist-test-slave-m9wg"
server_key: "37b3e93e90d88faea0c508a2edff0577"
server_key_iv: "eb2673148625bbe79bf5b7a92bd9fab3"
server_key_version: "encryptionkey@0"
I20260619 18:51:46.933096 22801 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 18:51:46.936213 22815 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:51:46.937393 22801 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 18:51:46.937546 22801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894810715282-4045-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894810715282-4045-0/minicluster-data/master-0/wal
uuid: "dc88dbc207444145a2234536d1da901c"
format_stamp: "Formatted at 2026-06-19 18:51:46 on dist-test-slave-m9wg"
server_key: "37b3e93e90d88faea0c508a2edff0577"
server_key_iv: "eb2673148625bbe79bf5b7a92bd9fab3"
server_key_version: "encryptionkey@0"
I20260619 18:51:46.937727 22801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894810715282-4045-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894810715282-4045-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894810715282-4045-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:51:46.957808 22801 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:51:46.958535 22801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:51:46.958781 22801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:51:46.974840 22801 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.126:44063
I20260619 18:51:46.974852 22866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.126:44063 every 8 connection(s)
I20260619 18:51:46.976006 22801 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
I20260619 18:51:46.979308 22867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:51:46.981330 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 22801
I20260619 18:51:46.981488 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894810715282-4045-0/minicluster-data/master-0/wal/instance
I20260619 18:51:46.981909 4045 external_mini_cluster.cc:1428] Setting key 1d99c314baf2a5848aef2288c7d52f5d
I20260619 18:51:46.991796 22867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc88dbc207444145a2234536d1da901c: Bootstrap starting.
I20260619 18:51:46.995301 22867 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc88dbc207444145a2234536d1da901c: Neither blocks nor log segments found. Creating new log.
I20260619 18:51:46.996258 22870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:46.983616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50254 (local address 127.3.243.126:44063)
0619 18:51:46.984630 (+ 1014us) server_negotiation.cc:206] Beginning negotiation
0619 18:51:46.984644 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 18:51:46.984691 (+ 47us) server_negotiation.cc:407] Connection header received
0619 18:51:46.986928 (+ 2237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:51:46.986953 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:51:46.987471 (+ 518us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:51:46.987853 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:51:46.988395 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:46.990281 (+ 1886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:46.991561 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:46.992553 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:46.992721 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:46.993746 (+ 1025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:51:46.993756 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:51:46.993773 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:51:46.994410 (+ 637us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:51:46.994522 (+ 112us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:51:46.994548 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:51:46.994653 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 18:51:46.994779 (+ 126us) server_negotiation.cc:299] Negotiation successful
0619 18:51:46.995062 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":297,"threads_started":1}
I20260619 18:51:46.996567 22867 log.cc:824] T 00000000000000000000000000000000 P dc88dbc207444145a2234536d1da901c: Log is configured to *not* fsync() on all Append() calls
I20260619 18:51:46.999410 22867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc88dbc207444145a2234536d1da901c: No bootstrap required, opened a new log
I20260619 18:51:47.006626 22867 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc88dbc207444145a2234536d1da901c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc88dbc207444145a2234536d1da901c" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 44063 } }
I20260619 18:51:47.006959 22867 raft_consensus.cc:374] T 00000000000000000000000000000000 P dc88dbc207444145a2234536d1da901c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:51:47.007041 22867 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc88dbc207444145a2234536d1da901c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc88dbc207444145a2234536d1da901c, State: Initialized, Role: FOLLOWER
I20260619 18:51:47.007534 22867 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc88dbc207444145a2234536d1da901c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc88dbc207444145a2234536d1da901c" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 44063 } }
I20260619 18:51:47.007735 22867 raft_consensus.cc:388] T 00000000000000000000000000000000 P dc88dbc207444145a2234536d1da901c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:51:47.007869 22867 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc88dbc207444145a2234536d1da901c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:51:47.008001 22867 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc88dbc207444145a2234536d1da901c [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:51:47.010293 22867 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc88dbc207444145a2234536d1da901c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc88dbc207444145a2234536d1da901c" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 44063 } }
I20260619 18:51:47.010715 22867 leader_election.cc:302] T 00000000000000000000000000000000 P dc88dbc207444145a2234536d1da901c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dc88dbc207444145a2234536d1da901c; no voters:
I20260619 18:51:47.011226 22867 leader_election.cc:288] T 00000000000000000000000000000000 P dc88dbc207444145a2234536d1da901c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:51:47.011329 22872 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc88dbc207444145a2234536d1da901c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:51:47.011919 22872 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc88dbc207444145a2234536d1da901c [term 1 LEADER]: Becoming Leader. State: Replica: dc88dbc207444145a2234536d1da901c, State: Running, Role: LEADER
I20260619 18:51:47.012432 22872 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc88dbc207444145a2234536d1da901c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc88dbc207444145a2234536d1da901c" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 44063 } }
I20260619 18:51:47.013085 22867 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc88dbc207444145a2234536d1da901c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:51:47.014882 22874 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc88dbc207444145a2234536d1da901c [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc88dbc207444145a2234536d1da901c. Latest consensus state: current_term: 1 leader_uuid: "dc88dbc207444145a2234536d1da901c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc88dbc207444145a2234536d1da901c" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 44063 } } }
I20260619 18:51:47.015285 22874 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc88dbc207444145a2234536d1da901c [sys.catalog]: This master's current role is: LEADER
I20260619 18:51:47.014906 22873 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc88dbc207444145a2234536d1da901c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dc88dbc207444145a2234536d1da901c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc88dbc207444145a2234536d1da901c" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 44063 } } }
I20260619 18:51:47.016045 22873 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc88dbc207444145a2234536d1da901c [sys.catalog]: This master's current role is: LEADER
I20260619 18:51:47.016470 22879 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:51:47.020924 22879 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:51:47.021621 22889 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 18:51:47.028203 22879 catalog_manager.cc:1347] Generated new cluster ID: 51c93700e3d3401688c69f127b4609e7
I20260619 18:51:47.028297 22879 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:51:47.043974 22879 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:51:47.045120 22879 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:51:47.053635 22879 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dc88dbc207444145a2234536d1da901c: Generated new TSK 0
I20260619 18:51:47.054335 22879 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:51:47.060667 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894810715282-4045-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894810715282-4045-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894810715282-4045-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894810715282-4045-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.65:0
--local_ip_for_outbound_sockets=127.3.243.65
--webserver_interface=127.3.243.65
--webserver_port=0
--tserver_master_addrs=127.3.243.126:44063
--builtin_ntp_servers=127.3.243.84:39311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 18:51:47.367363 22891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:51:47.367646 22891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:51:47.367780 22891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:51:47.375331 22891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:51:47.375443 22891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:51:47.375600 22891 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.65
I20260619 18:51:47.384697 22891 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:39311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894810715282-4045-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894810715282-4045-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894810715282-4045-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.243.65
--webserver_port=0
--tserver_master_addrs=127.3.243.126:44063
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.65
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894810715282-4045-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:51:47.385911 22891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:51:47.387643 22891 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:51:47.400348 22896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:51:47.400367 22897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:51:47.400279 22899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:51:47.401317 22891 server_base.cc:1034] running on GCE node
I20260619 18:51:47.401877 22891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:51:47.402740 22891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:51:47.403932 22891 hybrid_clock.cc:648] HybridClock initialized: now 1781895107403891 us; error 30 us; skew 500 ppm
I20260619 18:51:47.404265 22891 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 18:51:47.406349 22891 webserver.cc:460] Webserver started at http://127.3.243.65:40575/ using document root <none> and password file <none>
I20260619 18:51:47.406988 22891 fs_manager.cc:359] Metadata directory not provided
I20260619 18:51:47.407073 22891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:51:47.407331 22891 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:51:47.410040 22891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894810715282-4045-0/minicluster-data/ts-0/data/instance:
uuid: "4a23ffb46561499a805d8ca117ddd9bf"
format_stamp: "Formatted at 2026-06-19 18:51:47 on dist-test-slave-m9wg"
server_key: "07feb82a6e6d799141e38581d2ed9bcb"
server_key_iv: "4cb429e9e8fe2e1d3710f00a3a7cb334"
server_key_version: "encryptionkey@0"
I20260619 18:51:47.410737 22891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894810715282-4045-0/minicluster-data/ts-0/wal/instance:
uuid: "4a23ffb46561499a805d8ca117ddd9bf"
format_stamp: "Formatted at 2026-06-19 18:51:47 on dist-test-slave-m9wg"
server_key: "07feb82a6e6d799141e38581d2ed9bcb"
server_key_iv: "4cb429e9e8fe2e1d3710f00a3a7cb334"
server_key_version: "encryptionkey@0"
I20260619 18:51:47.415474 22891 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260619 18:51:47.418648 22905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:51:47.419816 22891 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 18:51:47.419951 22891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894810715282-4045-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894810715282-4045-0/minicluster-data/ts-0/wal
uuid: "4a23ffb46561499a805d8ca117ddd9bf"
format_stamp: "Formatted at 2026-06-19 18:51:47 on dist-test-slave-m9wg"
server_key: "07feb82a6e6d799141e38581d2ed9bcb"
server_key_iv: "4cb429e9e8fe2e1d3710f00a3a7cb334"
server_key_version: "encryptionkey@0"
I20260619 18:51:47.420112 22891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894810715282-4045-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894810715282-4045-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894810715282-4045-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:51:47.441493 22891 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:51:47.442236 22891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:51:47.442482 22891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:51:47.443320 22891 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:51:47.444671 22891 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:51:47.444756 22891 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:51:47.444828 22891 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:51:47.444888 22891 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:51:47.473238 22891 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.65:46511
I20260619 18:51:47.473259 23017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.65:46511 every 8 connection(s)
I20260619 18:51:47.474447 22891 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894810715282-4045-0/minicluster-data/ts-0/data/info.pb
I20260619 18:51:47.481743 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 22891
I20260619 18:51:47.481930 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894810715282-4045-0/minicluster-data/ts-0/wal/instance
I20260619 18:51:47.482503 4045 external_mini_cluster.cc:1428] Setting key 2dd49200444753bb6bc9afabf8c7b1e1
I20260619 18:51:47.485348 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894810715282-4045-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894810715282-4045-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894810715282-4045-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894810715282-4045-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.66:0
--local_ip_for_outbound_sockets=127.3.243.66
--webserver_interface=127.3.243.66
--webserver_port=0
--tserver_master_addrs=127.3.243.126:44063
--builtin_ntp_servers=127.3.243.84:39311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 18:51:47.486380 22870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:47.476724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.65:44203 (local address 127.3.243.126:44063)
0619 18:51:47.476920 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 18:51:47.476926 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:51:47.478332 (+ 1406us) server_negotiation.cc:407] Connection header received
0619 18:51:47.479937 (+ 1605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:51:47.479941 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:51:47.480004 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:51:47.480090 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:51:47.481027 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:47.481700 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:47.483684 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:47.484305 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:47.484443 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:47.485766 (+ 1323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:51:47.485770 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:51:47.485773 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:51:47.485790 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:51:47.485820 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:51:47.485831 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:51:47.485904 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 18:51:47.486092 (+ 188us) server_negotiation.cc:299] Negotiation successful
0619 18:51:47.486162 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 18:51:47.487596 23020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:47.477282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:44063 (local address 127.3.243.65:44203)
0619 18:51:47.478169 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:47.478208 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 18:51:47.479720 (+ 1512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:51:47.480284 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:51:47.480293 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:51:47.480381 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:51:47.480905 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:51:47.480920 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:51:47.481849 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:51:47.481855 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:51:47.483557 (+ 1702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:51:47.483568 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:51:47.484521 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:51:47.484528 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:51:47.484792 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:47.485455 (+ 663us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:51:47.485458 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:51:47.485508 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:51:47.485598 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:51:47.485911 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:51:47.485921 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:51:47.485928 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 18:51:47.486088 (+ 160us) client_negotiation.cc:240] Negotiation successful
0619 18:51:47.486442 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":247,"threads_started":1}
I20260619 18:51:47.488914 23018 heartbeater.cc:347] Connected to a master server at 127.3.243.126:44063
I20260619 18:51:47.489184 23018 heartbeater.cc:464] Registering TS with master...
I20260619 18:51:47.489945 23018 heartbeater.cc:511] Master 127.3.243.126:44063 requested a full tablet report, sending...
I20260619 18:51:47.491796 22832 ts_manager.cc:194] Registered new tserver with Master: 4a23ffb46561499a805d8ca117ddd9bf (127.3.243.65:46511)
I20260619 18:51:47.493309 22832 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.243.65:44203
W20260619 18:51:47.793682 23021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:51:47.793947 23021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:51:47.794067 23021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:51:47.801265 23021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:51:47.801357 23021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:51:47.801499 23021 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.66
I20260619 18:51:47.810153 23021 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:39311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894810715282-4045-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894810715282-4045-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894810715282-4045-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.243.66
--webserver_port=0
--tserver_master_addrs=127.3.243.126:44063
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.66
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894810715282-4045-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:51:47.811299 23021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:51:47.812963 23021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:51:47.825425 23026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:51:47.825387 23029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:51:47.825461 23027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:51:47.826292 23021 server_base.cc:1034] running on GCE node
I20260619 18:51:47.826908 23021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:51:47.827775 23021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:51:47.828966 23021 hybrid_clock.cc:648] HybridClock initialized: now 1781895107828918 us; error 37 us; skew 500 ppm
I20260619 18:51:47.829247 23021 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 18:51:47.831231 23021 webserver.cc:460] Webserver started at http://127.3.243.66:37585/ using document root <none> and password file <none>
I20260619 18:51:47.831851 23021 fs_manager.cc:359] Metadata directory not provided
I20260619 18:51:47.831947 23021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:51:47.832263 23021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:51:47.834883 23021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894810715282-4045-0/minicluster-data/ts-1/data/instance:
uuid: "e4fedcd8537f4229b86e1ea400ac58af"
format_stamp: "Formatted at 2026-06-19 18:51:47 on dist-test-slave-m9wg"
server_key: "510d22e51ee7549b499bdf580a8ca336"
server_key_iv: "dd6316aef7bd0d9455ad73eaba0aff71"
server_key_version: "encryptionkey@0"
I20260619 18:51:47.835595 23021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894810715282-4045-0/minicluster-data/ts-1/wal/instance:
uuid: "e4fedcd8537f4229b86e1ea400ac58af"
format_stamp: "Formatted at 2026-06-19 18:51:47 on dist-test-slave-m9wg"
server_key: "510d22e51ee7549b499bdf580a8ca336"
server_key_iv: "dd6316aef7bd0d9455ad73eaba0aff71"
server_key_version: "encryptionkey@0"
I20260619 18:51:47.840185 23021 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260619 18:51:47.843627 23035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:51:47.844882 23021 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 18:51:47.845026 23021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894810715282-4045-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894810715282-4045-0/minicluster-data/ts-1/wal
uuid: "e4fedcd8537f4229b86e1ea400ac58af"
format_stamp: "Formatted at 2026-06-19 18:51:47 on dist-test-slave-m9wg"
server_key: "510d22e51ee7549b499bdf580a8ca336"
server_key_iv: "dd6316aef7bd0d9455ad73eaba0aff71"
server_key_version: "encryptionkey@0"
I20260619 18:51:47.845223 23021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894810715282-4045-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894810715282-4045-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894810715282-4045-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:51:47.858783 23021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:51:47.859486 23021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:51:47.859743 23021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:51:47.860621 23021 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:51:47.861900 23021 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:51:47.861999 23021 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:51:47.862102 23021 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:51:47.862179 23021 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:51:47.890570 23021 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.66:43489
I20260619 18:51:47.890609 23147 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.66:43489 every 8 connection(s)
I20260619 18:51:47.891731 23021 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894810715282-4045-0/minicluster-data/ts-1/data/info.pb
I20260619 18:51:47.898175 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 23021
I20260619 18:51:47.898350 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894810715282-4045-0/minicluster-data/ts-1/wal/instance
I20260619 18:51:47.898898 4045 external_mini_cluster.cc:1428] Setting key 7b2708cf34cd7eb163b1f57220a6891c
I20260619 18:51:47.902061 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894810715282-4045-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894810715282-4045-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894810715282-4045-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894810715282-4045-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.67:0
--local_ip_for_outbound_sockets=127.3.243.67
--webserver_interface=127.3.243.67
--webserver_port=0
--tserver_master_addrs=127.3.243.126:44063
--builtin_ntp_servers=127.3.243.84:39311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 18:51:47.904721 22870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:47.894109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.66:44731 (local address 127.3.243.126:44063)
0619 18:51:47.894288 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 18:51:47.894292 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:51:47.895616 (+ 1324us) server_negotiation.cc:407] Connection header received
0619 18:51:47.897346 (+ 1730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:51:47.897352 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:51:47.897442 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:51:47.897535 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:51:47.898879 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:47.899649 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:47.902037 (+ 2388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:47.902531 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:47.902634 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:47.904035 (+ 1401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:51:47.904040 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:51:47.904043 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:51:47.904060 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:51:47.904092 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:51:47.904103 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:51:47.904220 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 18:51:47.904421 (+ 201us) server_negotiation.cc:299] Negotiation successful
0619 18:51:47.904498 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 18:51:47.905894 23150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:47.894612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:44063 (local address 127.3.243.66:44731)
0619 18:51:47.895467 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:47.895502 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 18:51:47.897084 (+ 1582us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:51:47.897776 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:51:47.897787 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:51:47.897902 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:51:47.898699 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:51:47.898726 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:51:47.899829 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:51:47.899834 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:51:47.901885 (+ 2051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:51:47.901903 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:51:47.902753 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:51:47.902761 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:51:47.903032 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:47.903724 (+ 692us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:51:47.903728 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:51:47.903778 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:51:47.903870 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:51:47.904227 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:51:47.904237 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:51:47.904244 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 18:51:47.904404 (+ 160us) client_negotiation.cc:240] Negotiation successful
0619 18:51:47.904748 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":271,"threads_started":1}
I20260619 18:51:47.907181 23148 heartbeater.cc:347] Connected to a master server at 127.3.243.126:44063
I20260619 18:51:47.907475 23148 heartbeater.cc:464] Registering TS with master...
I20260619 18:51:47.908322 23148 heartbeater.cc:511] Master 127.3.243.126:44063 requested a full tablet report, sending...
I20260619 18:51:47.909636 22832 ts_manager.cc:194] Registered new tserver with Master: e4fedcd8537f4229b86e1ea400ac58af (127.3.243.66:43489)
I20260619 18:51:47.910392 22832 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.243.66:44731
W20260619 18:51:48.208720 23151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:51:48.208956 23151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:51:48.209072 23151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:51:48.216013 23151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:51:48.216095 23151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:51:48.216255 23151 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.67
I20260619 18:51:48.224856 23151 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:39311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894810715282-4045-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894810715282-4045-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894810715282-4045-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.243.67
--webserver_port=0
--tserver_master_addrs=127.3.243.126:44063
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.67
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894810715282-4045-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:51:48.225956 23151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:51:48.227573 23151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:51:48.239282 23156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:51:48.239333 23157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:51:48.239310 23159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:51:48.239835 23151 server_base.cc:1034] running on GCE node
I20260619 18:51:48.240445 23151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:51:48.241255 23151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:51:48.242456 23151 hybrid_clock.cc:648] HybridClock initialized: now 1781895108242389 us; error 58 us; skew 500 ppm
I20260619 18:51:48.242731 23151 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 18:51:48.244728 23151 webserver.cc:460] Webserver started at http://127.3.243.67:32961/ using document root <none> and password file <none>
I20260619 18:51:48.245311 23151 fs_manager.cc:359] Metadata directory not provided
I20260619 18:51:48.245399 23151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:51:48.245677 23151 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:51:48.248258 23151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894810715282-4045-0/minicluster-data/ts-2/data/instance:
uuid: "4d5f431f24114a6ab4b86b09f910de67"
format_stamp: "Formatted at 2026-06-19 18:51:48 on dist-test-slave-m9wg"
server_key: "d49477b3ea87827ea699cade63523cfe"
server_key_iv: "01006680422336294af3a47da28df9f2"
server_key_version: "encryptionkey@0"
I20260619 18:51:48.248999 23151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894810715282-4045-0/minicluster-data/ts-2/wal/instance:
uuid: "4d5f431f24114a6ab4b86b09f910de67"
format_stamp: "Formatted at 2026-06-19 18:51:48 on dist-test-slave-m9wg"
server_key: "d49477b3ea87827ea699cade63523cfe"
server_key_iv: "01006680422336294af3a47da28df9f2"
server_key_version: "encryptionkey@0"
I20260619 18:51:48.253507 23151 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 18:51:48.256533 23165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:51:48.257659 23151 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260619 18:51:48.257805 23151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894810715282-4045-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894810715282-4045-0/minicluster-data/ts-2/wal
uuid: "4d5f431f24114a6ab4b86b09f910de67"
format_stamp: "Formatted at 2026-06-19 18:51:48 on dist-test-slave-m9wg"
server_key: "d49477b3ea87827ea699cade63523cfe"
server_key_iv: "01006680422336294af3a47da28df9f2"
server_key_version: "encryptionkey@0"
I20260619 18:51:48.257987 23151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894810715282-4045-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894810715282-4045-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894810715282-4045-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:51:48.280558 23151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:51:48.281306 23151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:51:48.281571 23151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:51:48.282426 23151 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:51:48.283741 23151 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:51:48.283840 23151 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:51:48.283938 23151 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:51:48.284008 23151 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:51:48.312129 23151 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.67:43719
I20260619 18:51:48.312227 23277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.67:43719 every 8 connection(s)
I20260619 18:51:48.313339 23151 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894810715282-4045-0/minicluster-data/ts-2/data/info.pb
I20260619 18:51:48.314796 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 23151
I20260619 18:51:48.314987 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894810715282-4045-0/minicluster-data/ts-2/wal/instance
I20260619 18:51:48.315546 4045 external_mini_cluster.cc:1428] Setting key febe5d99c0ada8548cb3e0f4497816d4
I20260619 18:51:48.324872 22870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:48.315981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.67:58617 (local address 127.3.243.126:44063)
0619 18:51:48.316220 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0619 18:51:48.316227 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:51:48.317786 (+ 1559us) server_negotiation.cc:407] Connection header received
0619 18:51:48.319497 (+ 1711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:51:48.319501 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:51:48.319564 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:51:48.319641 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:51:48.320596 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:48.321282 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:48.322684 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:48.323051 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:48.323136 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:48.324264 (+ 1128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:51:48.324268 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:51:48.324272 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:51:48.324289 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:51:48.324320 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:51:48.324332 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:51:48.324401 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 18:51:48.324577 (+ 176us) server_negotiation.cc:299] Negotiation successful
0619 18:51:48.324650 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 18:51:48.325708 23280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:48.316618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:44063 (local address 127.3.243.67:58617)
0619 18:51:48.317629 (+ 1011us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:48.317671 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 18:51:48.319272 (+ 1601us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:51:48.319793 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:51:48.319800 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:51:48.319887 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:51:48.320452 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:51:48.320468 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:51:48.321395 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:51:48.321398 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:51:48.322584 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:51:48.322596 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:51:48.323198 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:51:48.323202 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:51:48.323421 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:48.323978 (+ 557us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:51:48.323980 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:51:48.324014 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:51:48.324084 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:51:48.324406 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:51:48.324412 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:51:48.324417 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 18:51:48.324556 (+ 139us) client_negotiation.cc:240] Negotiation successful
0619 18:51:48.324825 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":283,"threads_started":1}
I20260619 18:51:48.326823 23278 heartbeater.cc:347] Connected to a master server at 127.3.243.126:44063
I20260619 18:51:48.327080 23278 heartbeater.cc:464] Registering TS with master...
I20260619 18:51:48.327718 23278 heartbeater.cc:511] Master 127.3.243.126:44063 requested a full tablet report, sending...
I20260619 18:51:48.328984 22832 ts_manager.cc:194] Registered new tserver with Master: 4d5f431f24114a6ab4b86b09f910de67 (127.3.243.67:43719)
I20260619 18:51:48.329980 22832 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.243.67:58617
I20260619 18:51:48.330209 4045 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 18:51:48.340567 22870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:48.334576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50270 (local address 127.3.243.126:44063)
0619 18:51:48.334710 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 18:51:48.334716 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:51:48.335145 (+ 429us) server_negotiation.cc:407] Connection header received
0619 18:51:48.335293 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:51:48.335297 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:51:48.335361 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:51:48.335439 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:51:48.335955 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:48.336692 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:48.338035 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:48.338512 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:48.338653 (+ 141us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:48.339782 (+ 1129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:51:48.339786 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:51:48.339790 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:51:48.339806 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:51:48.339835 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:51:48.339846 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:51:48.339957 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 18:51:48.340185 (+ 228us) server_negotiation.cc:299] Negotiation successful
0619 18:51:48.340240 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 18:51:48.344010 22832 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50270:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:51:48.350572 22832 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:51:48.372905 23290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:48.361492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.66:43489 (local address 127.0.0.1:39538)
0619 18:51:48.362438 (+ 946us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:48.362476 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 18:51:48.362638 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:51:48.365447 (+ 2809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:51:48.365455 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:51:48.365489 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:51:48.365900 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:51:48.365913 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:51:48.367716 (+ 1803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:51:48.367721 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:51:48.369755 (+ 2034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:51:48.369767 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:51:48.371923 (+ 2156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:51:48.371930 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:51:48.372132 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:48.372215 (+ 83us) client_negotiation.cc:769] Sending connection context
0619 18:51:48.372352 (+ 137us) client_negotiation.cc:240] Negotiation successful
0619 18:51:48.372535 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":684,"mutex_wait_us":37,"thread_start_us":905,"threads_started":1}
I20260619 18:51:48.373191 23293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:48.362116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45282 (local address 127.3.243.67:43719)
0619 18:51:48.363813 (+ 1697us) server_negotiation.cc:206] Beginning negotiation
0619 18:51:48.363820 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:51:48.365646 (+ 1826us) server_negotiation.cc:407] Connection header received
0619 18:51:48.365743 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:51:48.365751 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:51:48.365970 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:51:48.366133 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:51:48.367046 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:48.368248 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:48.370666 (+ 2418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:48.372180 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:48.372300 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:48.372547 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0619 18:51:48.372694 (+ 147us) server_negotiation.cc:299] Negotiation successful
0619 18:51:48.372858 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1527,"thread_start_us":1591,"threads_started":1}
I20260619 18:51:48.373507 23291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:48.361699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39538 (local address 127.3.243.66:43489)
0619 18:51:48.364865 (+ 3166us) server_negotiation.cc:206] Beginning negotiation
0619 18:51:48.364874 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:51:48.364905 (+ 31us) server_negotiation.cc:407] Connection header received
0619 18:51:48.365004 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:51:48.365011 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:51:48.365226 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:51:48.365396 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:51:48.366079 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:48.367558 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:48.369933 (+ 2375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:48.371729 (+ 1796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:48.371871 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:48.372140 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0619 18:51:48.373014 (+ 874us) server_negotiation.cc:299] Negotiation successful
0619 18:51:48.373174 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2994,"thread_start_us":2868,"threads_started":1}
I20260619 18:51:48.373673 23294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:48.363404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.65:46511 (local address 127.0.0.1:33268)
0619 18:51:48.365643 (+ 2239us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:48.365656 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 18:51:48.365780 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:51:48.366417 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:51:48.366422 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:51:48.366444 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:51:48.367209 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:51:48.367222 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:51:48.369045 (+ 1823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:51:48.369050 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:51:48.371112 (+ 2062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:51:48.371124 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:51:48.372967 (+ 1843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:51:48.372973 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:51:48.373132 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:48.373159 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 18:51:48.373234 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 18:51:48.373344 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2123,"mutex_wait_us":9,"thread_start_us":1988,"threads_started":1}
I20260619 18:51:48.373981 23292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:48.361850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.67:43719 (local address 127.0.0.1:45282)
0619 18:51:48.364223 (+ 2373us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:48.364239 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 18:51:48.364370 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:51:48.366117 (+ 1747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:51:48.366121 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:51:48.366145 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:51:48.366866 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:51:48.366879 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:51:48.368359 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:51:48.368364 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:51:48.370442 (+ 2078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:51:48.370457 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:51:48.372368 (+ 1911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:51:48.372376 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:51:48.372545 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:48.372575 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 18:51:48.372651 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 18:51:48.372767 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2238,"thread_start_us":2096,"threads_started":1}
I20260619 18:51:48.374567 23295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:48.363606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33268 (local address 127.3.243.65:46511)
0619 18:51:48.365157 (+ 1551us) server_negotiation.cc:206] Beginning negotiation
0619 18:51:48.365166 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:51:48.365753 (+ 587us) server_negotiation.cc:407] Connection header received
0619 18:51:48.365936 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:51:48.365941 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:51:48.366128 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:51:48.366313 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:51:48.367415 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:48.368898 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:48.371275 (+ 2377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:48.372488 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:48.372600 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:48.372807 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0619 18:51:48.373899 (+ 1092us) server_negotiation.cc:299] Negotiation successful
0619 18:51:48.374066 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1390,"thread_start_us":1291,"threads_started":1}
I20260619 18:51:48.376983 23083 tablet_service.cc:1467] Processing CreateTablet for tablet c874f5de7581422aa6df5ee1fb1bcdc4 (DEFAULT_TABLE table=test-table [id=57d394617dd84f6fb7527259561b24da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:51:48.377532 23213 tablet_service.cc:1467] Processing CreateTablet for tablet c874f5de7581422aa6df5ee1fb1bcdc4 (DEFAULT_TABLE table=test-table [id=57d394617dd84f6fb7527259561b24da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:51:48.378100 23083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c874f5de7581422aa6df5ee1fb1bcdc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:51:48.378082 22953 tablet_service.cc:1467] Processing CreateTablet for tablet c874f5de7581422aa6df5ee1fb1bcdc4 (DEFAULT_TABLE table=test-table [id=57d394617dd84f6fb7527259561b24da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:51:48.378901 23213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c874f5de7581422aa6df5ee1fb1bcdc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:51:48.379375 22953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c874f5de7581422aa6df5ee1fb1bcdc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:51:48.388598 23296 tablet_bootstrap.cc:492] T c874f5de7581422aa6df5ee1fb1bcdc4 P e4fedcd8537f4229b86e1ea400ac58af: Bootstrap starting.
I20260619 18:51:48.391467 23296 tablet_bootstrap.cc:654] T c874f5de7581422aa6df5ee1fb1bcdc4 P e4fedcd8537f4229b86e1ea400ac58af: Neither blocks nor log segments found. Creating new log.
I20260619 18:51:48.392798 23297 tablet_bootstrap.cc:492] T c874f5de7581422aa6df5ee1fb1bcdc4 P 4a23ffb46561499a805d8ca117ddd9bf: Bootstrap starting.
I20260619 18:51:48.393283 23296 log.cc:824] T c874f5de7581422aa6df5ee1fb1bcdc4 P e4fedcd8537f4229b86e1ea400ac58af: Log is configured to *not* fsync() on all Append() calls
I20260619 18:51:48.393914 23298 tablet_bootstrap.cc:492] T c874f5de7581422aa6df5ee1fb1bcdc4 P 4d5f431f24114a6ab4b86b09f910de67: Bootstrap starting.
I20260619 18:51:48.395679 23296 tablet_bootstrap.cc:492] T c874f5de7581422aa6df5ee1fb1bcdc4 P e4fedcd8537f4229b86e1ea400ac58af: No bootstrap required, opened a new log
I20260619 18:51:48.395924 23296 ts_tablet_manager.cc:1397] T c874f5de7581422aa6df5ee1fb1bcdc4 P e4fedcd8537f4229b86e1ea400ac58af: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20260619 18:51:48.397117 23297 tablet_bootstrap.cc:654] T c874f5de7581422aa6df5ee1fb1bcdc4 P 4a23ffb46561499a805d8ca117ddd9bf: Neither blocks nor log segments found. Creating new log.
I20260619 18:51:48.397117 23298 tablet_bootstrap.cc:654] T c874f5de7581422aa6df5ee1fb1bcdc4 P 4d5f431f24114a6ab4b86b09f910de67: Neither blocks nor log segments found. Creating new log.
I20260619 18:51:48.398212 23297 log.cc:824] T c874f5de7581422aa6df5ee1fb1bcdc4 P 4a23ffb46561499a805d8ca117ddd9bf: Log is configured to *not* fsync() on all Append() calls
I20260619 18:51:48.398211 23298 log.cc:824] T c874f5de7581422aa6df5ee1fb1bcdc4 P 4d5f431f24114a6ab4b86b09f910de67: Log is configured to *not* fsync() on all Append() calls
I20260619 18:51:48.400911 23298 tablet_bootstrap.cc:492] T c874f5de7581422aa6df5ee1fb1bcdc4 P 4d5f431f24114a6ab4b86b09f910de67: No bootstrap required, opened a new log
I20260619 18:51:48.401165 23298 ts_tablet_manager.cc:1397] T c874f5de7581422aa6df5ee1fb1bcdc4 P 4d5f431f24114a6ab4b86b09f910de67: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.002s
I20260619 18:51:48.401835 23297 tablet_bootstrap.cc:492] T c874f5de7581422aa6df5ee1fb1bcdc4 P 4a23ffb46561499a805d8ca117ddd9bf: No bootstrap required, opened a new log
I20260619 18:51:48.402056 23297 ts_tablet_manager.cc:1397] T c874f5de7581422aa6df5ee1fb1bcdc4 P 4a23ffb46561499a805d8ca117ddd9bf: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.005s
I20260619 18:51:48.403820 23296 raft_consensus.cc:348] T c874f5de7581422aa6df5ee1fb1bcdc4 P e4fedcd8537f4229b86e1ea400ac58af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4fedcd8537f4229b86e1ea400ac58af" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 43489 } } peers { permanent_uuid: "4d5f431f24114a6ab4b86b09f910de67" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 43719 } } peers { permanent_uuid: "4a23ffb46561499a805d8ca117ddd9bf" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 46511 } }
I20260619 18:51:48.404175 23296 raft_consensus.cc:374] T c874f5de7581422aa6df5ee1fb1bcdc4 P e4fedcd8537f4229b86e1ea400ac58af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:51:48.404274 23296 raft_consensus.cc:729] T c874f5de7581422aa6df5ee1fb1bcdc4 P e4fedcd8537f4229b86e1ea400ac58af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4fedcd8537f4229b86e1ea400ac58af, State: Initialized, Role: FOLLOWER
I20260619 18:51:48.404798 23296 consensus_queue.cc:260] T c874f5de7581422aa6df5ee1fb1bcdc4 P e4fedcd8537f4229b86e1ea400ac58af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4fedcd8537f4229b86e1ea400ac58af" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 43489 } } peers { permanent_uuid: "4d5f431f24114a6ab4b86b09f910de67" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 43719 } } peers { permanent_uuid: "4a23ffb46561499a805d8ca117ddd9bf" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 46511 } }
I20260619 18:51:48.405792 23148 heartbeater.cc:503] Master 127.3.243.126:44063 was elected leader, sending a full tablet report...
I20260619 18:51:48.406410 23296 ts_tablet_manager.cc:1428] T c874f5de7581422aa6df5ee1fb1bcdc4 P e4fedcd8537f4229b86e1ea400ac58af: Time spent starting tablet: real 0.010s user 0.004s sys 0.007s
I20260619 18:51:48.410983 23298 raft_consensus.cc:348] T c874f5de7581422aa6df5ee1fb1bcdc4 P 4d5f431f24114a6ab4b86b09f910de67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4fedcd8537f4229b86e1ea400ac58af" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 43489 } } peers { permanent_uuid: "4d5f431f24114a6ab4b86b09f910de67" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 43719 } } peers { permanent_uuid: "4a23ffb46561499a805d8ca117ddd9bf" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 46511 } }
I20260619 18:51:48.411384 23298 raft_consensus.cc:374] T c874f5de7581422aa6df5ee1fb1bcdc4 P 4d5f431f24114a6ab4b86b09f910de67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:51:48.411481 23298 raft_consensus.cc:729] T c874f5de7581422aa6df5ee1fb1bcdc4 P 4d5f431f24114a6ab4b86b09f910de67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d5f431f24114a6ab4b86b09f910de67, State: Initialized, Role: FOLLOWER
I20260619 18:51:48.411731 23297 raft_consensus.cc:348] T c874f5de7581422aa6df5ee1fb1bcdc4 P 4a23ffb46561499a805d8ca117ddd9bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4fedcd8537f4229b86e1ea400ac58af" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 43489 } } peers { permanent_uuid: "4d5f431f24114a6ab4b86b09f910de67" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 43719 } } peers { permanent_uuid: "4a23ffb46561499a805d8ca117ddd9bf" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 46511 } }
I20260619 18:51:48.412103 23297 raft_consensus.cc:374] T c874f5de7581422aa6df5ee1fb1bcdc4 P 4a23ffb46561499a805d8ca117ddd9bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:51:48.412233 23297 raft_consensus.cc:729] T c874f5de7581422aa6df5ee1fb1bcdc4 P 4a23ffb46561499a805d8ca117ddd9bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a23ffb46561499a805d8ca117ddd9bf, State: Initialized, Role: FOLLOWER
I20260619 18:51:48.412056 23298 consensus_queue.cc:260] T c874f5de7581422aa6df5ee1fb1bcdc4 P 4d5f431f24114a6ab4b86b09f910de67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4fedcd8537f4229b86e1ea400ac58af" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 43489 } } peers { permanent_uuid: "4d5f431f24114a6ab4b86b09f910de67" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 43719 } } peers { permanent_uuid: "4a23ffb46561499a805d8ca117ddd9bf" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 46511 } }
I20260619 18:51:48.412773 23297 consensus_queue.cc:260] T c874f5de7581422aa6df5ee1fb1bcdc4 P 4a23ffb46561499a805d8ca117ddd9bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4fedcd8537f4229b86e1ea400ac58af" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 43489 } } peers { permanent_uuid: "4d5f431f24114a6ab4b86b09f910de67" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 43719 } } peers { permanent_uuid: "4a23ffb46561499a805d8ca117ddd9bf" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 46511 } }
I20260619 18:51:48.413048 23278 heartbeater.cc:503] Master 127.3.243.126:44063 was elected leader, sending a full tablet report...
I20260619 18:51:48.413823 23018 heartbeater.cc:503] Master 127.3.243.126:44063 was elected leader, sending a full tablet report...
I20260619 18:51:48.414347 23297 ts_tablet_manager.cc:1428] T c874f5de7581422aa6df5ee1fb1bcdc4 P 4a23ffb46561499a805d8ca117ddd9bf: Time spent starting tablet: real 0.012s user 0.009s sys 0.004s
I20260619 18:51:48.414563 23298 ts_tablet_manager.cc:1428] T c874f5de7581422aa6df5ee1fb1bcdc4 P 4d5f431f24114a6ab4b86b09f910de67: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
W20260619 18:51:48.476385 23019 tablet.cc:2369] T c874f5de7581422aa6df5ee1fb1bcdc4 P 4a23ffb46561499a805d8ca117ddd9bf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:51:48.504398 23304 raft_consensus.cc:482] T c874f5de7581422aa6df5ee1fb1bcdc4 P 4a23ffb46561499a805d8ca117ddd9bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:51:48.504685 23304 raft_consensus.cc:504] T c874f5de7581422aa6df5ee1fb1bcdc4 P 4a23ffb46561499a805d8ca117ddd9bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4fedcd8537f4229b86e1ea400ac58af" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 43489 } } peers { permanent_uuid: "4d5f431f24114a6ab4b86b09f910de67" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 43719 } } peers { permanent_uuid: "4a23ffb46561499a805d8ca117ddd9bf" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 46511 } }
I20260619 18:51:48.506356 23304 leader_election.cc:288] T c874f5de7581422aa6df5ee1fb1bcdc4 P 4a23ffb46561499a805d8ca117ddd9bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4fedcd8537f4229b86e1ea400ac58af (127.3.243.66:43489), 4d5f431f24114a6ab4b86b09f910de67 (127.3.243.67:43719)
I20260619 18:51:48.513803 23291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:48.506760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.65:48205 (local address 127.3.243.66:43489)
0619 18:51:48.506948 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 18:51:48.506955 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:51:48.507679 (+ 724us) server_negotiation.cc:407] Connection header received
0619 18:51:48.507853 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:51:48.507859 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:51:48.507934 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:51:48.508002 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:51:48.508731 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:48.509422 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:48.511705 (+ 2283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:48.512875 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:48.513029 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:48.513098 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 18:51:48.513437 (+ 339us) server_negotiation.cc:299] Negotiation successful
0619 18:51:48.513568 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 18:51:48.513906 23293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:48.506872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.65:58547 (local address 127.3.243.67:43719)
0619 18:51:48.507050 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 18:51:48.507056 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:51:48.507731 (+ 675us) server_negotiation.cc:407] Connection header received
0619 18:51:48.507895 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:51:48.507898 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:51:48.507959 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:51:48.508043 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:51:48.508713 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:48.509414 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:48.511632 (+ 2218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:48.512874 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:48.513030 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:48.513098 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 18:51:48.513562 (+ 464us) server_negotiation.cc:299] Negotiation successful
0619 18:51:48.513687 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 18:51:48.513945 23305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:48.506756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.66:43489 (local address 127.3.243.65:48205)
0619 18:51:48.507558 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:48.507574 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 18:51:48.507702 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:51:48.508045 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:51:48.508050 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:51:48.508073 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:51:48.508594 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:51:48.508603 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:51:48.509557 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:51:48.509561 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:51:48.511450 (+ 1889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:51:48.511467 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:51:48.513073 (+ 1606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:51:48.513080 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:51:48.513277 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:48.513305 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 18:51:48.513390 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 18:51:48.513520 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":669,"mutex_wait_us":88,"thread_start_us":455,"threads_started":1}
I20260619 18:51:48.514511 23306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:48.506770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.67:43719 (local address 127.3.243.65:58547)
0619 18:51:48.507615 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:48.507631 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 18:51:48.507754 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:51:48.508068 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:51:48.508073 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:51:48.508100 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:51:48.508537 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:51:48.508551 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:51:48.509536 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:51:48.509542 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:51:48.511423 (+ 1881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:51:48.511439 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:51:48.513221 (+ 1782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:51:48.513227 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:51:48.513385 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:48.513414 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 18:51:48.514080 (+ 666us) client_negotiation.cc:240] Negotiation successful
0619 18:51:48.514227 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":749,"mutex_wait_us":83,"thread_start_us":456,"threads_started":1}
I20260619 18:51:48.515120 23103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c874f5de7581422aa6df5ee1fb1bcdc4" candidate_uuid: "4a23ffb46561499a805d8ca117ddd9bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4fedcd8537f4229b86e1ea400ac58af" is_pre_election: true
I20260619 18:51:48.515579 23103 raft_consensus.cc:2454] T c874f5de7581422aa6df5ee1fb1bcdc4 P e4fedcd8537f4229b86e1ea400ac58af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a23ffb46561499a805d8ca117ddd9bf in term 0.
I20260619 18:51:48.516181 22908 leader_election.cc:302] T c874f5de7581422aa6df5ee1fb1bcdc4 P 4a23ffb46561499a805d8ca117ddd9bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a23ffb46561499a805d8ca117ddd9bf, e4fedcd8537f4229b86e1ea400ac58af; no voters:
I20260619 18:51:48.516098 23233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c874f5de7581422aa6df5ee1fb1bcdc4" candidate_uuid: "4a23ffb46561499a805d8ca117ddd9bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d5f431f24114a6ab4b86b09f910de67" is_pre_election: true
I20260619 18:51:48.516589 23304 raft_consensus.cc:2781] T c874f5de7581422aa6df5ee1fb1bcdc4 P 4a23ffb46561499a805d8ca117ddd9bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:51:48.516707 23304 raft_consensus.cc:482] T c874f5de7581422aa6df5ee1fb1bcdc4 P 4a23ffb46561499a805d8ca117ddd9bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:51:48.516707 23233 raft_consensus.cc:2454] T c874f5de7581422aa6df5ee1fb1bcdc4 P 4d5f431f24114a6ab4b86b09f910de67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a23ffb46561499a805d8ca117ddd9bf in term 0.
I20260619 18:51:48.516794 23304 raft_consensus.cc:3037] T c874f5de7581422aa6df5ee1fb1bcdc4 P 4a23ffb46561499a805d8ca117ddd9bf [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:51:48.519035 23304 raft_consensus.cc:504] T c874f5de7581422aa6df5ee1fb1bcdc4 P 4a23ffb46561499a805d8ca117ddd9bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4fedcd8537f4229b86e1ea400ac58af" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 43489 } } peers { permanent_uuid: "4d5f431f24114a6ab4b86b09f910de67" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 43719 } } peers { permanent_uuid: "4a23ffb46561499a805d8ca117ddd9bf" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 46511 } }
I20260619 18:51:48.519560 23304 leader_election.cc:288] T c874f5de7581422aa6df5ee1fb1bcdc4 P 4a23ffb46561499a805d8ca117ddd9bf [CANDIDATE]: Term 1 election: Requested vote from peers e4fedcd8537f4229b86e1ea400ac58af (127.3.243.66:43489), 4d5f431f24114a6ab4b86b09f910de67 (127.3.243.67:43719)
I20260619 18:51:48.519963 23103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c874f5de7581422aa6df5ee1fb1bcdc4" candidate_uuid: "4a23ffb46561499a805d8ca117ddd9bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4fedcd8537f4229b86e1ea400ac58af"
I20260619 18:51:48.520025 23233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c874f5de7581422aa6df5ee1fb1bcdc4" candidate_uuid: "4a23ffb46561499a805d8ca117ddd9bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d5f431f24114a6ab4b86b09f910de67"
I20260619 18:51:48.520124 23103 raft_consensus.cc:3037] T c874f5de7581422aa6df5ee1fb1bcdc4 P e4fedcd8537f4229b86e1ea400ac58af [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:51:48.520224 23233 raft_consensus.cc:3037] T c874f5de7581422aa6df5ee1fb1bcdc4 P 4d5f431f24114a6ab4b86b09f910de67 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:51:48.522238 23103 raft_consensus.cc:2454] T c874f5de7581422aa6df5ee1fb1bcdc4 P e4fedcd8537f4229b86e1ea400ac58af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a23ffb46561499a805d8ca117ddd9bf in term 1.
I20260619 18:51:48.522343 23233 raft_consensus.cc:2454] T c874f5de7581422aa6df5ee1fb1bcdc4 P 4d5f431f24114a6ab4b86b09f910de67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a23ffb46561499a805d8ca117ddd9bf in term 1.
I20260619 18:51:48.522565 22908 leader_election.cc:302] T c874f5de7581422aa6df5ee1fb1bcdc4 P 4a23ffb46561499a805d8ca117ddd9bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a23ffb46561499a805d8ca117ddd9bf, e4fedcd8537f4229b86e1ea400ac58af; no voters:
I20260619 18:51:48.522855 23304 raft_consensus.cc:2781] T c874f5de7581422aa6df5ee1fb1bcdc4 P 4a23ffb46561499a805d8ca117ddd9bf [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:51:48.523252 23304 raft_consensus.cc:686] T c874f5de7581422aa6df5ee1fb1bcdc4 P 4a23ffb46561499a805d8ca117ddd9bf [term 1 LEADER]: Becoming Leader. State: Replica: 4a23ffb46561499a805d8ca117ddd9bf, State: Running, Role: LEADER
I20260619 18:51:48.523749 23304 consensus_queue.cc:237] T c874f5de7581422aa6df5ee1fb1bcdc4 P 4a23ffb46561499a805d8ca117ddd9bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4fedcd8537f4229b86e1ea400ac58af" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 43489 } } peers { permanent_uuid: "4d5f431f24114a6ab4b86b09f910de67" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 43719 } } peers { permanent_uuid: "4a23ffb46561499a805d8ca117ddd9bf" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 46511 } }
I20260619 18:51:48.527849 22832 catalog_manager.cc:5495] T c874f5de7581422aa6df5ee1fb1bcdc4 P 4a23ffb46561499a805d8ca117ddd9bf reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a23ffb46561499a805d8ca117ddd9bf (127.3.243.65). New cstate: current_term: 1 leader_uuid: "4a23ffb46561499a805d8ca117ddd9bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4fedcd8537f4229b86e1ea400ac58af" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 43489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d5f431f24114a6ab4b86b09f910de67" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 43719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a23ffb46561499a805d8ca117ddd9bf" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 46511 } health_report { overall_health: HEALTHY } } }
I20260619 18:51:48.561261 23295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:48.556629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33270 (local address 127.3.243.65:46511)
0619 18:51:48.556799 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 18:51:48.556805 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:51:48.556821 (+ 16us) server_negotiation.cc:407] Connection header received
0619 18:51:48.556955 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:51:48.556958 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:51:48.557046 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:51:48.557155 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:51:48.557457 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:48.558121 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:48.559336 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:48.559711 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:48.559793 (+ 82us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:48.560227 (+ 434us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:51:48.560676 (+ 449us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:51:48.560810 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 18:51:48.560927 (+ 117us) server_negotiation.cc:299] Negotiation successful
0619 18:51:48.560965 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260619 18:51:48.564906 23279 tablet.cc:2369] T c874f5de7581422aa6df5ee1fb1bcdc4 P 4d5f431f24114a6ab4b86b09f910de67: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:51:48.568738 23233 raft_consensus.cc:1261] T c874f5de7581422aa6df5ee1fb1bcdc4 P 4d5f431f24114a6ab4b86b09f910de67 [term 1 FOLLOWER]: Refusing update from remote peer 4a23ffb46561499a805d8ca117ddd9bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:51:48.568830 23103 raft_consensus.cc:1261] T c874f5de7581422aa6df5ee1fb1bcdc4 P e4fedcd8537f4229b86e1ea400ac58af [term 1 FOLLOWER]: Refusing update from remote peer 4a23ffb46561499a805d8ca117ddd9bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:51:48.569648 23307 consensus_queue.cc:1035] T c874f5de7581422aa6df5ee1fb1bcdc4 P 4a23ffb46561499a805d8ca117ddd9bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d5f431f24114a6ab4b86b09f910de67" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 43719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:51:48.569947 23304 consensus_queue.cc:1035] T c874f5de7581422aa6df5ee1fb1bcdc4 P 4a23ffb46561499a805d8ca117ddd9bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4fedcd8537f4229b86e1ea400ac58af" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 43489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:51:48.579149 23310 mvcc.cc:205] Tried to move back new op lower bound from 7298642364689223680 to 7298642364517081088. Current Snapshot: MvccSnapshot[applied={T|T < 7298642364689223680}]
I20260619 18:51:48.580174 23311 mvcc.cc:205] Tried to move back new op lower bound from 7298642364689223680 to 7298642364517081088. Current Snapshot: MvccSnapshot[applied={T|T < 7298642364689223680}]
I20260619 18:51:48.583921 23312 mvcc.cc:205] Tried to move back new op lower bound from 7298642364689223680 to 7298642364517081088. Current Snapshot: MvccSnapshot[applied={T|T < 7298642364689223680}]
I20260619 18:51:48.592994 22832 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50270:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:51:48.593210 22832 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50270:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:51:48.596427 22832 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dc88dbc207444145a2234536d1da901c: Sending DeleteTablet for 3 replicas of tablet c874f5de7581422aa6df5ee1fb1bcdc4
I20260619 18:51:48.597455 22953 tablet_service.cc:1514] Processing DeleteTablet for tablet c874f5de7581422aa6df5ee1fb1bcdc4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:51:48 UTC) from {username='slave'} at 127.0.0.1:33268
I20260619 18:51:48.597455 23213 tablet_service.cc:1514] Processing DeleteTablet for tablet c874f5de7581422aa6df5ee1fb1bcdc4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:51:48 UTC) from {username='slave'} at 127.0.0.1:45282
I20260619 18:51:48.597496 23083 tablet_service.cc:1514] Processing DeleteTablet for tablet c874f5de7581422aa6df5ee1fb1bcdc4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:51:48 UTC) from {username='slave'} at 127.0.0.1:39538
I20260619 18:51:48.598290 23319 tablet_replica.cc:330] T c874f5de7581422aa6df5ee1fb1bcdc4 P 4d5f431f24114a6ab4b86b09f910de67: stopping tablet replica
I20260619 18:51:48.598685 23319 raft_consensus.cc:2229] T c874f5de7581422aa6df5ee1fb1bcdc4 P 4d5f431f24114a6ab4b86b09f910de67 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:51:48.599013 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 22891
I20260619 18:51:48.599134 23319 raft_consensus.cc:2258] T c874f5de7581422aa6df5ee1fb1bcdc4 P 4d5f431f24114a6ab4b86b09f910de67 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:51:48.599869 23320 tablet_replica.cc:330] T c874f5de7581422aa6df5ee1fb1bcdc4 P 4a23ffb46561499a805d8ca117ddd9bf: stopping tablet replica
I20260619 18:51:48.600869 23320 raft_consensus.cc:2229] T c874f5de7581422aa6df5ee1fb1bcdc4 P 4a23ffb46561499a805d8ca117ddd9bf [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:51:48.601509 23320 raft_consensus.cc:2258] T c874f5de7581422aa6df5ee1fb1bcdc4 P 4a23ffb46561499a805d8ca117ddd9bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:51:48.603600 23320 ts_tablet_manager.cc:1905] T c874f5de7581422aa6df5ee1fb1bcdc4 P 4a23ffb46561499a805d8ca117ddd9bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:51:48.609176 23319 ts_tablet_manager.cc:1905] T c874f5de7581422aa6df5ee1fb1bcdc4 P 4d5f431f24114a6ab4b86b09f910de67: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:51:48.612592 23321 tablet_replica.cc:330] T c874f5de7581422aa6df5ee1fb1bcdc4 P e4fedcd8537f4229b86e1ea400ac58af: stopping tablet replica
I20260619 18:51:48.613030 23321 raft_consensus.cc:2229] T c874f5de7581422aa6df5ee1fb1bcdc4 P e4fedcd8537f4229b86e1ea400ac58af [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:51:48.613543 23321 raft_consensus.cc:2258] T c874f5de7581422aa6df5ee1fb1bcdc4 P e4fedcd8537f4229b86e1ea400ac58af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:51:48.615515 23321 ts_tablet_manager.cc:1905] T c874f5de7581422aa6df5ee1fb1bcdc4 P e4fedcd8537f4229b86e1ea400ac58af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:51:48.615842 23320 ts_tablet_manager.cc:1918] T c874f5de7581422aa6df5ee1fb1bcdc4 P 4a23ffb46561499a805d8ca117ddd9bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:51:48.616037 23320 log.cc:1196] T c874f5de7581422aa6df5ee1fb1bcdc4 P 4a23ffb46561499a805d8ca117ddd9bf: Deleting WAL directory at /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894810715282-4045-0/minicluster-data/ts-0/wal/wals/c874f5de7581422aa6df5ee1fb1bcdc4
I20260619 18:51:48.616567 23319 ts_tablet_manager.cc:1918] T c874f5de7581422aa6df5ee1fb1bcdc4 P 4d5f431f24114a6ab4b86b09f910de67: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:51:48.616710 23319 log.cc:1196] T c874f5de7581422aa6df5ee1fb1bcdc4 P 4d5f431f24114a6ab4b86b09f910de67: Deleting WAL directory at /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894810715282-4045-0/minicluster-data/ts-2/wal/wals/c874f5de7581422aa6df5ee1fb1bcdc4
I20260619 18:51:48.617197 23319 ts_tablet_manager.cc:1939] T c874f5de7581422aa6df5ee1fb1bcdc4 P 4d5f431f24114a6ab4b86b09f910de67: Deleting consensus metadata
I20260619 18:51:48.617583 23320 ts_tablet_manager.cc:1939] T c874f5de7581422aa6df5ee1fb1bcdc4 P 4a23ffb46561499a805d8ca117ddd9bf: Deleting consensus metadata
I20260619 18:51:48.618891 22817 catalog_manager.cc:4841] TS 4d5f431f24114a6ab4b86b09f910de67 (127.3.243.67:43719): tablet c874f5de7581422aa6df5ee1fb1bcdc4 (table test-table [id=57d394617dd84f6fb7527259561b24da]) successfully deleted
I20260619 18:51:48.619817 23295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:48.601914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33286 (local address 127.3.243.65:46511)
0619 18:51:48.602081 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 18:51:48.602086 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:51:48.610754 (+ 8668us) server_negotiation.cc:407] Connection header received
0619 18:51:48.611172 (+ 418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:51:48.611178 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:51:48.611277 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:51:48.611360 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:51:48.612002 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:48.613119 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:48.615149 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:48.616042 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:48.616183 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:48.617585 (+ 1402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:51:48.617594 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:51:48.617609 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:51:48.618344 (+ 735us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:51:48.618396 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:51:48.618418 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:51:48.618489 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 18:51:48.619345 (+ 856us) server_negotiation.cc:299] Negotiation successful
0619 18:51:48.619449 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 18:51:48.620226 22819 catalog_manager.cc:4841] TS 4a23ffb46561499a805d8ca117ddd9bf (127.3.243.65:46511): tablet c874f5de7581422aa6df5ee1fb1bcdc4 (table test-table [id=57d394617dd84f6fb7527259561b24da]) successfully deleted
I20260619 18:51:48.621037 23013 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:51:48.621898 23321 ts_tablet_manager.cc:1918] T c874f5de7581422aa6df5ee1fb1bcdc4 P e4fedcd8537f4229b86e1ea400ac58af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:51:48.622068 23321 log.cc:1196] T c874f5de7581422aa6df5ee1fb1bcdc4 P e4fedcd8537f4229b86e1ea400ac58af: Deleting WAL directory at /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894810715282-4045-0/minicluster-data/ts-1/wal/wals/c874f5de7581422aa6df5ee1fb1bcdc4
I20260619 18:51:48.622530 23321 ts_tablet_manager.cc:1939] T c874f5de7581422aa6df5ee1fb1bcdc4 P e4fedcd8537f4229b86e1ea400ac58af: Deleting consensus metadata
I20260619 18:51:48.624027 22818 catalog_manager.cc:4841] TS e4fedcd8537f4229b86e1ea400ac58af (127.3.243.66:43489): tablet c874f5de7581422aa6df5ee1fb1bcdc4 (table test-table [id=57d394617dd84f6fb7527259561b24da]) successfully deleted
I20260619 18:51:48.718006 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 22891
I20260619 18:51:48.732277 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 23021
I20260619 18:51:48.738267 23291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:48.732995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39550 (local address 127.3.243.66:43489)
0619 18:51:48.733190 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 18:51:48.733194 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:51:48.733212 (+ 18us) server_negotiation.cc:407] Connection header received
0619 18:51:48.733308 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:51:48.733312 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:51:48.733399 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:51:48.733477 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:51:48.733816 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:48.734511 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:48.735707 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:48.736082 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:48.736285 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:48.737117 (+ 832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:51:48.737125 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:51:48.737140 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:51:48.737673 (+ 533us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:51:48.737718 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:51:48.737736 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:51:48.737827 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 18:51:48.737882 (+ 55us) server_negotiation.cc:299] Negotiation successful
0619 18:51:48.737961 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 18:51:48.738835 23143 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:51:48.826884 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 23021
I20260619 18:51:48.840519 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 23151
I20260619 18:51:48.846807 23293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:48.841362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45296 (local address 127.3.243.67:43719)
0619 18:51:48.841592 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0619 18:51:48.841597 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:51:48.841620 (+ 23us) server_negotiation.cc:407] Connection header received
0619 18:51:48.841712 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:51:48.841718 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:51:48.841849 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:51:48.841942 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:51:48.842292 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:48.842995 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:48.844138 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:48.844610 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:48.844695 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:48.845654 (+ 959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:51:48.845663 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:51:48.845678 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:51:48.846246 (+ 568us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:51:48.846294 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:51:48.846314 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:51:48.846395 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 18:51:48.846485 (+ 90us) server_negotiation.cc:299] Negotiation successful
0619 18:51:48.846570 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 18:51:48.847411 23273 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:51:48.936628 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 23151
I20260619 18:51:48.950402 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 22801
I20260619 18:51:48.951470 22862 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:51:49.060173 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 22801
2026-06-19T18:51:49Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2542 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894810715282-4045-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:51:49 dist-test-slave-m9wg krb5kdc[23329](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:51:49 dist-test-slave-m9wg krb5kdc[23329](info): set up 2 sockets
Jun 19 18:51:49 dist-test-slave-m9wg krb5kdc[23329](info): commencing operation
krb5kdc: starting...
W20260619 18:51:50.084072 4045 mini_kdc.cc:121] Time spent starting KDC: real 0.988s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894810715282-4045-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894810715282-4045-0/krb5kdc/test-user.keytab.
Jun 19 18:51:50 dist-test-slave-m9wg krb5kdc[23329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895110, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:51:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:51:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.243.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab.
Entry for principal kudu/127.3.243.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab.
I20260619 18:51:50.300972 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894810715282-4045-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894810715282-4045-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894810715282-4045-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.126:34521
--webserver_interface=127.3.243.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab
--principal=kudu/127.3.243.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:43639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.126:34521
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:51:50.604863 23342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:51:50.605122 23342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:51:50.605249 23342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:51:50.612016 23342 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:51:50.612121 23342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:51:50.612215 23342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:51:50.612296 23342 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:51:50.612358 23342 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:51:50.621519 23342 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:43639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894810715282-4045-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894810715282-4045-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.243.126:34521
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab
--principal=kudu/127.3.243.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.126:34521
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.243.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894810715282-4045-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:51:50.622692 23342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:51:50.623937 23342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:51:50.634893 23348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:51:50.634975 23347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:51:50.635560 23350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:51:50.638047 23342 server_base.cc:1034] running on GCE node
I20260619 18:51:50.638871 23342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:51:50.640380 23342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:51:50.641605 23342 hybrid_clock.cc:648] HybridClock initialized: now 1781895110641540 us; error 45 us; skew 500 ppm
I20260619 18:51:50.641964 23342 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:51:50 dist-test-slave-m9wg krb5kdc[23329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895110, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:51:50.644770 23342 init.cc:377] Logged in from keytab as kudu/127.3.243.126@KRBTEST.COM (short username kudu)
I20260619 18:51:50.646101 23342 webserver.cc:460] Webserver started at http://127.3.243.126:45421/ using document root <none> and password file <none>
I20260619 18:51:50.646751 23342 fs_manager.cc:359] Metadata directory not provided
I20260619 18:51:50.646871 23342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:51:50.647148 23342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:51:50.650040 23342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894810715282-4045-0/minicluster-data/master-0/data/instance:
uuid: "0f54603dcc0b449b8a46dc40d1e3e533"
format_stamp: "Formatted at 2026-06-19 18:51:50 on dist-test-slave-m9wg"
server_key: "4deabdf482cdd97346ededf07e74202d"
server_key_iv: "96c89ce0fcd1d23defff94c5e41edfb2"
server_key_version: "encryptionkey@0"
I20260619 18:51:50.650731 23342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894810715282-4045-0/minicluster-data/master-0/wal/instance:
uuid: "0f54603dcc0b449b8a46dc40d1e3e533"
format_stamp: "Formatted at 2026-06-19 18:51:50 on dist-test-slave-m9wg"
server_key: "4deabdf482cdd97346ededf07e74202d"
server_key_iv: "96c89ce0fcd1d23defff94c5e41edfb2"
server_key_version: "encryptionkey@0"
I20260619 18:51:50.655548 23342 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 18:51:50.658793 23357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:51:50.660324 23342 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:51:50.660459 23342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894810715282-4045-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894810715282-4045-0/minicluster-data/master-0/wal
uuid: "0f54603dcc0b449b8a46dc40d1e3e533"
format_stamp: "Formatted at 2026-06-19 18:51:50 on dist-test-slave-m9wg"
server_key: "4deabdf482cdd97346ededf07e74202d"
server_key_iv: "96c89ce0fcd1d23defff94c5e41edfb2"
server_key_version: "encryptionkey@0"
I20260619 18:51:50.660616 23342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894810715282-4045-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894810715282-4045-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894810715282-4045-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:51:50.675104 23342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:51:50.678622 23342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:51:50.678843 23342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:51:50.694867 23342 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.126:34521
I20260619 18:51:50.694885 23408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.126:34521 every 8 connection(s)
I20260619 18:51:50.696102 23342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
I20260619 18:51:50.698386 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 23342
I20260619 18:51:50.698532 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894810715282-4045-0/minicluster-data/master-0/wal/instance
I20260619 18:51:50.698933 4045 external_mini_cluster.cc:1428] Setting key 67c097dea8e7f3596cc7c7da545e0a07
I20260619 18:51:50.699895 23409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 18:51:50 dist-test-slave-m9wg krb5kdc[23329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895110, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.126@KRBTEST.COM
I20260619 18:51:50.712543 23409 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f54603dcc0b449b8a46dc40d1e3e533: Bootstrap starting.
I20260619 18:51:50.716142 23409 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f54603dcc0b449b8a46dc40d1e3e533: Neither blocks nor log segments found. Creating new log.
I20260619 18:51:50.717339 23409 log.cc:824] T 00000000000000000000000000000000 P 0f54603dcc0b449b8a46dc40d1e3e533: Log is configured to *not* fsync() on all Append() calls
I20260619 18:51:50.720389 23409 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f54603dcc0b449b8a46dc40d1e3e533: No bootstrap required, opened a new log
I20260619 18:51:50.721081 23412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:50.700587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34170 (local address 127.3.243.126:34521)
0619 18:51:50.701349 (+ 762us) server_negotiation.cc:206] Beginning negotiation
0619 18:51:50.701363 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 18:51:50.701408 (+ 45us) server_negotiation.cc:407] Connection header received
0619 18:51:50.703122 (+ 1714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:51:50.703148 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:51:50.703610 (+ 462us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:51:50.704018 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:51:50.705263 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:50.707183 (+ 1920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:50.708412 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:50.709373 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:50.709560 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:50.713286 (+ 3726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:51:50.713296 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:51:50.713312 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:51:50.713355 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:51:50.716925 (+ 3570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:51:50.717584 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:51:50.717592 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:51:50.717600 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:51:50.717817 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:51:50.718227 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:51:50.718230 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:51:50.718232 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:51:50.718807 (+ 575us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:51:50.719064 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0619 18:51:50.719444 (+ 380us) server_negotiation.cc:299] Negotiation successful
0619 18:51:50.719836 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":301,"threads_started":1}
I20260619 18:51:50.727481 23409 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f54603dcc0b449b8a46dc40d1e3e533 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f54603dcc0b449b8a46dc40d1e3e533" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 34521 } }
I20260619 18:51:50.727769 23409 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0f54603dcc0b449b8a46dc40d1e3e533 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:51:50.727836 23409 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f54603dcc0b449b8a46dc40d1e3e533 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f54603dcc0b449b8a46dc40d1e3e533, State: Initialized, Role: FOLLOWER
I20260619 18:51:50.728338 23409 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f54603dcc0b449b8a46dc40d1e3e533 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f54603dcc0b449b8a46dc40d1e3e533" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 34521 } }
I20260619 18:51:50.728528 23409 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0f54603dcc0b449b8a46dc40d1e3e533 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:51:50.728614 23409 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f54603dcc0b449b8a46dc40d1e3e533 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:51:50.728731 23409 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f54603dcc0b449b8a46dc40d1e3e533 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:51:50.730800 23409 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f54603dcc0b449b8a46dc40d1e3e533 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f54603dcc0b449b8a46dc40d1e3e533" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 34521 } }
I20260619 18:51:50.731194 23409 leader_election.cc:302] T 00000000000000000000000000000000 P 0f54603dcc0b449b8a46dc40d1e3e533 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0f54603dcc0b449b8a46dc40d1e3e533; no voters:
I20260619 18:51:50.731673 23409 leader_election.cc:288] T 00000000000000000000000000000000 P 0f54603dcc0b449b8a46dc40d1e3e533 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:51:50.731772 23414 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f54603dcc0b449b8a46dc40d1e3e533 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:51:50.732393 23414 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0f54603dcc0b449b8a46dc40d1e3e533 [term 1 LEADER]: Becoming Leader. State: Replica: 0f54603dcc0b449b8a46dc40d1e3e533, State: Running, Role: LEADER
I20260619 18:51:50.732859 23414 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f54603dcc0b449b8a46dc40d1e3e533 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f54603dcc0b449b8a46dc40d1e3e533" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 34521 } }
I20260619 18:51:50.733397 23409 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f54603dcc0b449b8a46dc40d1e3e533 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:51:50.735275 23415 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f54603dcc0b449b8a46dc40d1e3e533 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0f54603dcc0b449b8a46dc40d1e3e533" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f54603dcc0b449b8a46dc40d1e3e533" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 34521 } } }
I20260619 18:51:50.735616 23415 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f54603dcc0b449b8a46dc40d1e3e533 [sys.catalog]: This master's current role is: LEADER
I20260619 18:51:50.736601 23416 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f54603dcc0b449b8a46dc40d1e3e533 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f54603dcc0b449b8a46dc40d1e3e533. Latest consensus state: current_term: 1 leader_uuid: "0f54603dcc0b449b8a46dc40d1e3e533" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f54603dcc0b449b8a46dc40d1e3e533" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 34521 } } }
I20260619 18:51:50.736954 23416 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f54603dcc0b449b8a46dc40d1e3e533 [sys.catalog]: This master's current role is: LEADER
I20260619 18:51:50.737404 23419 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:51:50.742180 23419 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:51:50.750556 23419 catalog_manager.cc:1347] Generated new cluster ID: a7b70099ff2a488682d33f11f6d4164c
I20260619 18:51:50.750680 23419 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:51:50.788731 23419 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:51:50.789980 23419 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:51:50.799368 23419 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0f54603dcc0b449b8a46dc40d1e3e533: Generated new TSK 0
I20260619 18:51:50.800140 23419 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.243.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.65.keytab.
Entry for principal kudu/127.3.243.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.65.keytab.
I20260619 18:51:50.868848 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894810715282-4045-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894810715282-4045-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894810715282-4045-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894810715282-4045-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.65:0
--local_ip_for_outbound_sockets=127.3.243.65
--webserver_interface=127.3.243.65
--webserver_port=0
--tserver_master_addrs=127.3.243.126:34521
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.65.keytab
--principal=kudu/127.3.243.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:43639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:51:51.179687 23435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:51:51.179944 23435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:51:51.180065 23435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:51:51.187078 23435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:51:51.187201 23435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:51:51.187353 23435 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.65
I20260619 18:51:51.196887 23435 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:43639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894810715282-4045-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894810715282-4045-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.65.keytab
--principal=kudu/127.3.243.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894810715282-4045-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.243.65
--webserver_port=0
--tserver_master_addrs=127.3.243.126:34521
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.65
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894810715282-4045-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:51:51.198061 23435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:51:51.199313 23435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:51:51.211283 23443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:51:51.211241 23441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:51:51.211334 23440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:51:51.212127 23435 server_base.cc:1034] running on GCE node
I20260619 18:51:51.212705 23435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:51:51.213567 23435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:51:51.214766 23435 hybrid_clock.cc:648] HybridClock initialized: now 1781895111214712 us; error 50 us; skew 500 ppm
I20260619 18:51:51.215065 23435 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:51:51 dist-test-slave-m9wg krb5kdc[23329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895111, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:51:51.218058 23435 init.cc:377] Logged in from keytab as kudu/127.3.243.65@KRBTEST.COM (short username kudu)
I20260619 18:51:51.219344 23435 webserver.cc:460] Webserver started at http://127.3.243.65:43221/ using document root <none> and password file <none>
I20260619 18:51:51.220067 23435 fs_manager.cc:359] Metadata directory not provided
I20260619 18:51:51.220213 23435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:51:51.220546 23435 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:51:51.223598 23435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894810715282-4045-0/minicluster-data/ts-0/data/instance:
uuid: "10747c43ea7247b796a05f6e41f21e95"
format_stamp: "Formatted at 2026-06-19 18:51:51 on dist-test-slave-m9wg"
server_key: "b751095f0c16f65d4cd5011f108be1c6"
server_key_iv: "7435e79015eb13096ec8351300947070"
server_key_version: "encryptionkey@0"
I20260619 18:51:51.224342 23435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894810715282-4045-0/minicluster-data/ts-0/wal/instance:
uuid: "10747c43ea7247b796a05f6e41f21e95"
format_stamp: "Formatted at 2026-06-19 18:51:51 on dist-test-slave-m9wg"
server_key: "b751095f0c16f65d4cd5011f108be1c6"
server_key_iv: "7435e79015eb13096ec8351300947070"
server_key_version: "encryptionkey@0"
I20260619 18:51:51.228940 23435 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 18:51:51.232048 23450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:51:51.233254 23435 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 18:51:51.233374 23435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894810715282-4045-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894810715282-4045-0/minicluster-data/ts-0/wal
uuid: "10747c43ea7247b796a05f6e41f21e95"
format_stamp: "Formatted at 2026-06-19 18:51:51 on dist-test-slave-m9wg"
server_key: "b751095f0c16f65d4cd5011f108be1c6"
server_key_iv: "7435e79015eb13096ec8351300947070"
server_key_version: "encryptionkey@0"
I20260619 18:51:51.233528 23435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894810715282-4045-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894810715282-4045-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894810715282-4045-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:51:51.257292 23435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:51:51.261154 23435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:51:51.261415 23435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:51:51.262193 23435 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:51:51.263365 23435 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:51:51.263434 23435 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:51:51.263562 23435 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:51:51.263612 23435 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:51:51.292063 23435 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.65:34191
I20260619 18:51:51.292105 23562 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.65:34191 every 8 connection(s)
I20260619 18:51:51.293277 23435 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894810715282-4045-0/minicluster-data/ts-0/data/info.pb
I20260619 18:51:51.302182 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 23435
I20260619 18:51:51.302366 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894810715282-4045-0/minicluster-data/ts-0/wal/instance
I20260619 18:51:51.302906 4045 external_mini_cluster.cc:1428] Setting key 9d7b2375263cdc7766ff2b353aa1cbec
Jun 19 18:51:51 dist-test-slave-m9wg krb5kdc[23329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895111, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.65@KRBTEST.COM for kudu/127.3.243.126@KRBTEST.COM
I20260619 18:51:51.313234 23565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:51.295633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.65:44345 (local address 127.3.243.126:34521)
0619 18:51:51.296050 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0619 18:51:51.296055 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:51:51.297027 (+ 972us) server_negotiation.cc:407] Connection header received
0619 18:51:51.298858 (+ 1831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:51:51.298864 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:51:51.298953 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:51:51.299084 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:51:51.300662 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:51.301370 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:51.303451 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:51.303976 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:51.304072 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:51.307948 (+ 3876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:51:51.307952 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:51:51.307956 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:51:51.307985 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:51:51.310348 (+ 2363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:51:51.311117 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:51:51.311121 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:51:51.311124 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:51:51.311254 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:51:51.311668 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:51:51.311671 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:51:51.311673 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:51:51.311919 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:51:51.312022 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 18:51:51.312809 (+ 787us) server_negotiation.cc:299] Negotiation successful
0619 18:51:51.312971 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":203,"threads_started":1}
I20260619 18:51:51.314265 23566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:51.296207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:34521 (local address 127.3.243.65:44345)
0619 18:51:51.296872 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:51.296915 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 18:51:51.298581 (+ 1666us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:51:51.299306 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:51:51.299323 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:51:51.299891 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:51:51.300498 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:51:51.300515 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:51:51.301520 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:51:51.301526 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:51:51.303296 (+ 1770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:51:51.303329 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:51:51.304208 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:51:51.304216 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:51:51.304498 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:51.305161 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:51:51.305191 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:51:51.307740 (+ 2549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:51:51.310470 (+ 2730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:51:51.310478 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:51:51.310493 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:51:51.310998 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:51:51.311352 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:51:51.311356 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:51:51.311359 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:51:51.311580 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:51:51.312030 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:51:51.312037 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:51:51.312444 (+ 407us) client_negotiation.cc:769] Sending connection context
0619 18:51:51.312715 (+ 271us) client_negotiation.cc:240] Negotiation successful
0619 18:51:51.313198 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":234,"threads_started":1}
I20260619 18:51:51.315713 23563 heartbeater.cc:347] Connected to a master server at 127.3.243.126:34521
I20260619 18:51:51.315995 23563 heartbeater.cc:464] Registering TS with master...
I20260619 18:51:51.316699 23563 heartbeater.cc:511] Master 127.3.243.126:34521 requested a full tablet report, sending...
I20260619 18:51:51.318570 23374 ts_manager.cc:194] Registered new tserver with Master: 10747c43ea7247b796a05f6e41f21e95 (127.3.243.65:34191)
I20260619 18:51:51.319932 23374 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.65@KRBTEST.COM'} at 127.3.243.65:44345
WARNING: no policy specified for kudu/127.3.243.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.66.keytab.
Entry for principal kudu/127.3.243.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.66.keytab.
I20260619 18:51:51.350711 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894810715282-4045-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894810715282-4045-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894810715282-4045-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894810715282-4045-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.66:0
--local_ip_for_outbound_sockets=127.3.243.66
--webserver_interface=127.3.243.66
--webserver_port=0
--tserver_master_addrs=127.3.243.126:34521
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.66.keytab
--principal=kudu/127.3.243.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:43639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:51:51.652024 23569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:51:51.652295 23569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:51:51.652417 23569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:51:51.659314 23569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:51:51.659407 23569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:51:51.659557 23569 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.66
I20260619 18:51:51.668896 23569 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:43639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894810715282-4045-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894810715282-4045-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.66.keytab
--principal=kudu/127.3.243.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894810715282-4045-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.243.66
--webserver_port=0
--tserver_master_addrs=127.3.243.126:34521
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.66
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894810715282-4045-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:51:51.670028 23569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:51:51.671221 23569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:51:51.683185 23577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:51:51.683145 23575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:51:51.683348 23574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:51:51.683878 23569 server_base.cc:1034] running on GCE node
I20260619 18:51:51.684443 23569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:51:51.685262 23569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:51:51.686452 23569 hybrid_clock.cc:648] HybridClock initialized: now 1781895111686404 us; error 41 us; skew 500 ppm
I20260619 18:51:51.686803 23569 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:51:51 dist-test-slave-m9wg krb5kdc[23329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895111, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:51:51.689699 23569 init.cc:377] Logged in from keytab as kudu/127.3.243.66@KRBTEST.COM (short username kudu)
I20260619 18:51:51.690986 23569 webserver.cc:460] Webserver started at http://127.3.243.66:40137/ using document root <none> and password file <none>
I20260619 18:51:51.691625 23569 fs_manager.cc:359] Metadata directory not provided
I20260619 18:51:51.691732 23569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:51:51.692039 23569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:51:51.694928 23569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894810715282-4045-0/minicluster-data/ts-1/data/instance:
uuid: "0047bd8659e148a3926c148821a8b350"
format_stamp: "Formatted at 2026-06-19 18:51:51 on dist-test-slave-m9wg"
server_key: "f058eca82ee441ed3a4045e81d698ca6"
server_key_iv: "3502eaa771e80a9ee8a3d85ecf1cacfc"
server_key_version: "encryptionkey@0"
I20260619 18:51:51.695777 23569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894810715282-4045-0/minicluster-data/ts-1/wal/instance:
uuid: "0047bd8659e148a3926c148821a8b350"
format_stamp: "Formatted at 2026-06-19 18:51:51 on dist-test-slave-m9wg"
server_key: "f058eca82ee441ed3a4045e81d698ca6"
server_key_iv: "3502eaa771e80a9ee8a3d85ecf1cacfc"
server_key_version: "encryptionkey@0"
I20260619 18:51:51.700773 23569 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 18:51:51.704088 23584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:51:51.705401 23569 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 18:51:51.705536 23569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894810715282-4045-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894810715282-4045-0/minicluster-data/ts-1/wal
uuid: "0047bd8659e148a3926c148821a8b350"
format_stamp: "Formatted at 2026-06-19 18:51:51 on dist-test-slave-m9wg"
server_key: "f058eca82ee441ed3a4045e81d698ca6"
server_key_iv: "3502eaa771e80a9ee8a3d85ecf1cacfc"
server_key_version: "encryptionkey@0"
I20260619 18:51:51.705710 23569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894810715282-4045-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894810715282-4045-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894810715282-4045-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:51:51.729369 23569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:51:51.732976 23569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:51:51.733196 23569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:51:51.734117 23569 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:51:51.735690 23569 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:51:51.735785 23569 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:51:51.735903 23569 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:51:51.735963 23569 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:51:51.764356 23569 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.66:39919
I20260619 18:51:51.764395 23696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.66:39919 every 8 connection(s)
I20260619 18:51:51.765636 23569 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894810715282-4045-0/minicluster-data/ts-1/data/info.pb
I20260619 18:51:51.769584 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 23569
I20260619 18:51:51.769780 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894810715282-4045-0/minicluster-data/ts-1/wal/instance
I20260619 18:51:51.770299 4045 external_mini_cluster.cc:1428] Setting key da72c68204ce6bc7106a6fc23743a68c
Jun 19 18:51:51 dist-test-slave-m9wg krb5kdc[23329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895111, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.66@KRBTEST.COM for kudu/127.3.243.126@KRBTEST.COM
I20260619 18:51:51.786067 23565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:51.767920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.66:51899 (local address 127.3.243.126:34521)
0619 18:51:51.768120 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 18:51:51.768126 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:51:51.769210 (+ 1084us) server_negotiation.cc:407] Connection header received
0619 18:51:51.771528 (+ 2318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:51:51.771534 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:51:51.771628 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:51:51.771728 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:51:51.773529 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:51.774247 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:51.776210 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:51.776603 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:51.776691 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:51.780489 (+ 3798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:51:51.780494 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:51:51.780497 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:51:51.780530 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:51:51.783232 (+ 2702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:51:51.783932 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:51:51.783937 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:51:51.783941 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:51:51.784088 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:51:51.784483 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:51:51.784488 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:51:51.784492 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:51:51.784806 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:51:51.784952 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 18:51:51.785537 (+ 585us) server_negotiation.cc:299] Negotiation successful
0619 18:51:51.785744 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 18:51:51.786881 23699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:51.768397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:34521 (local address 127.3.243.66:51899)
0619 18:51:51.769055 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:51.769099 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 18:51:51.771252 (+ 2153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:51:51.771909 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:51:51.771917 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:51:51.772625 (+ 708us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:51:51.773367 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:51:51.773392 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:51:51.774376 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:51:51.774381 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:51:51.776057 (+ 1676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:51:51.776077 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:51:51.776793 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:51:51.776800 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:51:51.777056 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:51.777692 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:51:51.777719 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:51:51.780251 (+ 2532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:51:51.783360 (+ 3109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:51:51.783367 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:51:51.783379 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:51:51.783794 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:51:51.784195 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:51:51.784200 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:51:51.784204 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:51:51.784384 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:51:51.784906 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:51:51.784911 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:51:51.785208 (+ 297us) client_negotiation.cc:769] Sending connection context
0619 18:51:51.785419 (+ 211us) client_negotiation.cc:240] Negotiation successful
0619 18:51:51.785786 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":243,"threads_started":1}
I20260619 18:51:51.788220 23697 heartbeater.cc:347] Connected to a master server at 127.3.243.126:34521
I20260619 18:51:51.788522 23697 heartbeater.cc:464] Registering TS with master...
I20260619 18:51:51.789258 23697 heartbeater.cc:511] Master 127.3.243.126:34521 requested a full tablet report, sending...
I20260619 18:51:51.790747 23374 ts_manager.cc:194] Registered new tserver with Master: 0047bd8659e148a3926c148821a8b350 (127.3.243.66:39919)
I20260619 18:51:51.791543 23374 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.66@KRBTEST.COM'} at 127.3.243.66:51899
WARNING: no policy specified for kudu/127.3.243.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.67.keytab.
Entry for principal kudu/127.3.243.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.67.keytab.
I20260619 18:51:51.825263 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894810715282-4045-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894810715282-4045-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894810715282-4045-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894810715282-4045-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.67:0
--local_ip_for_outbound_sockets=127.3.243.67
--webserver_interface=127.3.243.67
--webserver_port=0
--tserver_master_addrs=127.3.243.126:34521
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.67.keytab
--principal=kudu/127.3.243.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:43639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:51:52.128008 23702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:51:52.128324 23702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:51:52.128466 23702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:51:52.135588 23702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:51:52.135689 23702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:51:52.135840 23702 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.67
I20260619 18:51:52.145409 23702 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:43639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894810715282-4045-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894810715282-4045-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.67.keytab
--principal=kudu/127.3.243.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894810715282-4045-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.243.67
--webserver_port=0
--tserver_master_addrs=127.3.243.126:34521
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.67
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894810715282-4045-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:51:52.146639 23702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:51:52.147913 23702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:51:52.160348 23710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:51:52.160356 23708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:51:52.160466 23707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:51:52.160953 23702 server_base.cc:1034] running on GCE node
I20260619 18:51:52.161537 23702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:51:52.162410 23702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:51:52.163602 23702 hybrid_clock.cc:648] HybridClock initialized: now 1781895112163558 us; error 41 us; skew 500 ppm
I20260619 18:51:52.163937 23702 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:51:52 dist-test-slave-m9wg krb5kdc[23329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895112, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:51:52.166879 23702 init.cc:377] Logged in from keytab as kudu/127.3.243.67@KRBTEST.COM (short username kudu)
I20260619 18:51:52.168301 23702 webserver.cc:460] Webserver started at http://127.3.243.67:37201/ using document root <none> and password file <none>
I20260619 18:51:52.168949 23702 fs_manager.cc:359] Metadata directory not provided
I20260619 18:51:52.169060 23702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:51:52.169344 23702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:51:52.172207 23702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894810715282-4045-0/minicluster-data/ts-2/data/instance:
uuid: "f8b13e3a547d4aa4976a23a7ec31e93b"
format_stamp: "Formatted at 2026-06-19 18:51:52 on dist-test-slave-m9wg"
server_key: "66db06dfef8fd0c36d6865c16c2bf145"
server_key_iv: "19b15c2568bd1245e2663a1c23686d6a"
server_key_version: "encryptionkey@0"
I20260619 18:51:52.172958 23702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894810715282-4045-0/minicluster-data/ts-2/wal/instance:
uuid: "f8b13e3a547d4aa4976a23a7ec31e93b"
format_stamp: "Formatted at 2026-06-19 18:51:52 on dist-test-slave-m9wg"
server_key: "66db06dfef8fd0c36d6865c16c2bf145"
server_key_iv: "19b15c2568bd1245e2663a1c23686d6a"
server_key_version: "encryptionkey@0"
I20260619 18:51:52.177832 23702 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 18:51:52.181066 23717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:51:52.182287 23702 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 18:51:52.182492 23702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894810715282-4045-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894810715282-4045-0/minicluster-data/ts-2/wal
uuid: "f8b13e3a547d4aa4976a23a7ec31e93b"
format_stamp: "Formatted at 2026-06-19 18:51:52 on dist-test-slave-m9wg"
server_key: "66db06dfef8fd0c36d6865c16c2bf145"
server_key_iv: "19b15c2568bd1245e2663a1c23686d6a"
server_key_version: "encryptionkey@0"
I20260619 18:51:52.182665 23702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894810715282-4045-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894810715282-4045-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894810715282-4045-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:51:52.200045 23702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:51:52.203692 23702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:51:52.203943 23702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:51:52.204870 23702 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:51:52.206128 23702 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:51:52.206202 23702 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:51:52.206314 23702 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:51:52.206364 23702 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:51:52.235359 23702 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.67:35417
I20260619 18:51:52.235457 23829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.67:35417 every 8 connection(s)
I20260619 18:51:52.236682 23702 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894810715282-4045-0/minicluster-data/ts-2/data/info.pb
I20260619 18:51:52.245850 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 23702
I20260619 18:51:52.246031 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894810715282-4045-0/minicluster-data/ts-2/wal/instance
I20260619 18:51:52.246572 4045 external_mini_cluster.cc:1428] Setting key 4cf12cf5c5a5fae947424feb4601db6f
Jun 19 18:51:52 dist-test-slave-m9wg krb5kdc[23329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895112, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.67@KRBTEST.COM for kudu/127.3.243.126@KRBTEST.COM
I20260619 18:51:52.255416 23565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:52.239142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.67:53389 (local address 127.3.243.126:34521)
0619 18:51:52.239321 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 18:51:52.239326 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:51:52.240537 (+ 1211us) server_negotiation.cc:407] Connection header received
0619 18:51:52.242306 (+ 1769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:51:52.242313 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:51:52.242392 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:51:52.242479 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:51:52.243955 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:52.244721 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:52.246662 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:52.247143 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:52.247259 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:52.250735 (+ 3476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:51:52.250740 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:51:52.250743 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:51:52.250775 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:51:52.253026 (+ 2251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:51:52.253618 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:51:52.253621 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:51:52.253623 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:51:52.253727 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:51:52.254093 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:51:52.254097 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:51:52.254099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:51:52.254366 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:51:52.254474 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 18:51:52.255023 (+ 549us) server_negotiation.cc:299] Negotiation successful
0619 18:51:52.255179 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 18:51:52.256385 23832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:52.239642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:34521 (local address 127.3.243.67:53389)
0619 18:51:52.240397 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:52.240442 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 18:51:52.242045 (+ 1603us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:51:52.242722 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:51:52.242730 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:51:52.243236 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:51:52.243786 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:51:52.243804 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:51:52.244873 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:51:52.244876 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:51:52.246518 (+ 1642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:51:52.246532 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:51:52.247371 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:51:52.247379 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:51:52.247619 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:52.248314 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:51:52.248341 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:51:52.250550 (+ 2209us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:51:52.253134 (+ 2584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:51:52.253140 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:51:52.253152 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:51:52.253532 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:51:52.253825 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:51:52.253828 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:51:52.253830 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:51:52.253990 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:51:52.254451 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:51:52.254456 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:51:52.254732 (+ 276us) client_negotiation.cc:769] Sending connection context
0619 18:51:52.254954 (+ 222us) client_negotiation.cc:240] Negotiation successful
0619 18:51:52.255302 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":250,"threads_started":1}
I20260619 18:51:52.257613 23830 heartbeater.cc:347] Connected to a master server at 127.3.243.126:34521
I20260619 18:51:52.257856 23830 heartbeater.cc:464] Registering TS with master...
I20260619 18:51:52.258498 23830 heartbeater.cc:511] Master 127.3.243.126:34521 requested a full tablet report, sending...
I20260619 18:51:52.259779 23374 ts_manager.cc:194] Registered new tserver with Master: f8b13e3a547d4aa4976a23a7ec31e93b (127.3.243.67:35417)
I20260619 18:51:52.260633 23374 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.67@KRBTEST.COM'} at 127.3.243.67:53389
I20260619 18:51:52.261555 4045 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 18:51:52.276494 23565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:52.265451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34178 (local address 127.3.243.126:34521)
0619 18:51:52.265593 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 18:51:52.265597 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:51:52.266014 (+ 417us) server_negotiation.cc:407] Connection header received
0619 18:51:52.266132 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:51:52.266135 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:51:52.266195 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:51:52.266266 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:51:52.267648 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:52.268371 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:52.269545 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:52.269921 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:52.270002 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:52.272033 (+ 2031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:51:52.272037 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:51:52.272040 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:51:52.272069 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:51:52.274283 (+ 2214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:51:52.274879 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:51:52.274882 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:51:52.274884 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:51:52.275001 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:51:52.275292 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:51:52.275295 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:51:52.275297 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:51:52.275594 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:51:52.275767 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0619 18:51:52.276058 (+ 291us) server_negotiation.cc:299] Negotiation successful
0619 18:51:52.276259 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 18:51:52.280278 23373 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34178:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:51:52.286916 23373 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:51:52.317790 23844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:52.298470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53454 (local address 127.3.243.65:34191)
0619 18:51:52.309226 (+ 10756us) server_negotiation.cc:206] Beginning negotiation
0619 18:51:52.309234 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:51:52.309262 (+ 28us) server_negotiation.cc:407] Connection header received
0619 18:51:52.309373 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:51:52.309381 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:51:52.309627 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:51:52.310606 (+ 979us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:51:52.310817 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:52.312515 (+ 1698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:52.315318 (+ 2803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:52.316652 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:52.316789 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:52.317008 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0619 18:51:52.317297 (+ 289us) server_negotiation.cc:299] Negotiation successful
0619 18:51:52.317462 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10599,"thread_start_us":10459,"threads_started":1}
I20260619 18:51:52.318120 23843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:52.298162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.65:34191 (local address 127.0.0.1:53454)
0619 18:51:52.305088 (+ 6926us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:52.305139 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 18:51:52.305326 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:51:52.309921 (+ 4595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:51:52.309930 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:51:52.309964 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:51:52.310410 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:51:52.310423 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:51:52.312670 (+ 2247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:51:52.312676 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:51:52.315120 (+ 2444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:51:52.315133 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:51:52.316850 (+ 1717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:51:52.316856 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:51:52.317051 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:52.317104 (+ 53us) client_negotiation.cc:769] Sending connection context
0619 18:51:52.317230 (+ 126us) client_negotiation.cc:240] Negotiation successful
0619 18:51:52.317405 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6655,"mutex_wait_us":1,"thread_start_us":11026,"threads_started":1}
I20260619 18:51:52.318600 23842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:52.298322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.67:35417 (local address 127.0.0.1:54700)
0619 18:51:52.309414 (+ 11092us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:52.309430 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 18:51:52.309540 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:51:52.310436 (+ 896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:51:52.310442 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:51:52.310475 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:51:52.310889 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:51:52.310901 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:51:52.313057 (+ 2156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:51:52.313067 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:51:52.315831 (+ 2764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:51:52.315844 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:51:52.317784 (+ 1940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:51:52.317791 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:51:52.317962 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:52.317997 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 18:51:52.318084 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 18:51:52.318205 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10987,"mutex_wait_us":1,"thread_start_us":6351,"threads_started":1}
I20260619 18:51:52.318923 23845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:52.298908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54700 (local address 127.3.243.67:35417)
0619 18:51:52.308920 (+ 10012us) server_negotiation.cc:206] Beginning negotiation
0619 18:51:52.308930 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:51:52.309528 (+ 598us) server_negotiation.cc:407] Connection header received
0619 18:51:52.309673 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:51:52.309682 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:51:52.309943 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:51:52.310140 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:51:52.311115 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:52.312883 (+ 1768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:52.315979 (+ 3096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:52.317297 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:52.317414 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:52.317623 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0619 18:51:52.318443 (+ 820us) server_negotiation.cc:299] Negotiation successful
0619 18:51:52.318608 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9811,"thread_start_us":9677,"threads_started":1}
I20260619 18:51:52.321960 23498 tablet_service.cc:1467] Processing CreateTablet for tablet 97f0c503682846019df0fe41b6baa067 (DEFAULT_TABLE table=test-table [id=03520df56e4f4378b2f7fcb9ca148d0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:51:52.322292 23765 tablet_service.cc:1467] Processing CreateTablet for tablet 97f0c503682846019df0fe41b6baa067 (DEFAULT_TABLE table=test-table [id=03520df56e4f4378b2f7fcb9ca148d0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:51:52.323174 23847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:52.309754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.66:39919 (local address 127.0.0.1:33936)
0619 18:51:52.314862 (+ 5108us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:52.314879 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 18:51:52.314997 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:51:52.315626 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:51:52.315631 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:51:52.315655 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:51:52.316052 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:51:52.316060 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:51:52.317627 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:51:52.317632 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:51:52.320507 (+ 2875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:51:52.320523 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:51:52.322475 (+ 1952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:51:52.322481 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:51:52.322638 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:52.322669 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 18:51:52.322754 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 18:51:52.322885 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4967,"thread_start_us":4823,"threads_started":1}
I20260619 18:51:52.323457 23498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97f0c503682846019df0fe41b6baa067. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:51:52.323632 23765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97f0c503682846019df0fe41b6baa067. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:51:52.323855 23846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:52.305768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33936 (local address 127.3.243.66:39919)
0619 18:51:52.314519 (+ 8751us) server_negotiation.cc:206] Beginning negotiation
0619 18:51:52.314527 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:51:52.315021 (+ 494us) server_negotiation.cc:407] Connection header received
0619 18:51:52.315113 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:51:52.315120 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:51:52.315345 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:51:52.315516 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:51:52.316206 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:52.317464 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:52.320672 (+ 3208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:52.322025 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:52.322158 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:52.322326 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0619 18:51:52.323389 (+ 1063us) server_negotiation.cc:299] Negotiation successful
0619 18:51:52.323549 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8598,"thread_start_us":8505,"threads_started":1}
I20260619 18:51:52.327123 23563 heartbeater.cc:503] Master 127.3.243.126:34521 was elected leader, sending a full tablet report...
I20260619 18:51:52.327157 23632 tablet_service.cc:1467] Processing CreateTablet for tablet 97f0c503682846019df0fe41b6baa067 (DEFAULT_TABLE table=test-table [id=03520df56e4f4378b2f7fcb9ca148d0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:51:52.328442 23632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97f0c503682846019df0fe41b6baa067. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:51:52.335175 23848 tablet_bootstrap.cc:492] T 97f0c503682846019df0fe41b6baa067 P 10747c43ea7247b796a05f6e41f21e95: Bootstrap starting.
I20260619 18:51:52.337029 23849 tablet_bootstrap.cc:492] T 97f0c503682846019df0fe41b6baa067 P f8b13e3a547d4aa4976a23a7ec31e93b: Bootstrap starting.
I20260619 18:51:52.340322 23848 tablet_bootstrap.cc:654] T 97f0c503682846019df0fe41b6baa067 P 10747c43ea7247b796a05f6e41f21e95: Neither blocks nor log segments found. Creating new log.
I20260619 18:51:52.340332 23849 tablet_bootstrap.cc:654] T 97f0c503682846019df0fe41b6baa067 P f8b13e3a547d4aa4976a23a7ec31e93b: Neither blocks nor log segments found. Creating new log.
I20260619 18:51:52.340644 23850 tablet_bootstrap.cc:492] T 97f0c503682846019df0fe41b6baa067 P 0047bd8659e148a3926c148821a8b350: Bootstrap starting.
I20260619 18:51:52.341632 23849 log.cc:824] T 97f0c503682846019df0fe41b6baa067 P f8b13e3a547d4aa4976a23a7ec31e93b: Log is configured to *not* fsync() on all Append() calls
I20260619 18:51:52.341873 23848 log.cc:824] T 97f0c503682846019df0fe41b6baa067 P 10747c43ea7247b796a05f6e41f21e95: Log is configured to *not* fsync() on all Append() calls
I20260619 18:51:52.343442 23850 tablet_bootstrap.cc:654] T 97f0c503682846019df0fe41b6baa067 P 0047bd8659e148a3926c148821a8b350: Neither blocks nor log segments found. Creating new log.
I20260619 18:51:52.344465 23850 log.cc:824] T 97f0c503682846019df0fe41b6baa067 P 0047bd8659e148a3926c148821a8b350: Log is configured to *not* fsync() on all Append() calls
I20260619 18:51:52.344527 23849 tablet_bootstrap.cc:492] T 97f0c503682846019df0fe41b6baa067 P f8b13e3a547d4aa4976a23a7ec31e93b: No bootstrap required, opened a new log
I20260619 18:51:52.344781 23849 ts_tablet_manager.cc:1397] T 97f0c503682846019df0fe41b6baa067 P f8b13e3a547d4aa4976a23a7ec31e93b: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.004s
I20260619 18:51:52.345295 23848 tablet_bootstrap.cc:492] T 97f0c503682846019df0fe41b6baa067 P 10747c43ea7247b796a05f6e41f21e95: No bootstrap required, opened a new log
I20260619 18:51:52.345583 23848 ts_tablet_manager.cc:1397] T 97f0c503682846019df0fe41b6baa067 P 10747c43ea7247b796a05f6e41f21e95: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.003s
I20260619 18:51:52.346848 23850 tablet_bootstrap.cc:492] T 97f0c503682846019df0fe41b6baa067 P 0047bd8659e148a3926c148821a8b350: No bootstrap required, opened a new log
I20260619 18:51:52.347111 23850 ts_tablet_manager.cc:1397] T 97f0c503682846019df0fe41b6baa067 P 0047bd8659e148a3926c148821a8b350: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260619 18:51:52.353832 23850 raft_consensus.cc:348] T 97f0c503682846019df0fe41b6baa067 P 0047bd8659e148a3926c148821a8b350 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10747c43ea7247b796a05f6e41f21e95" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 34191 } } peers { permanent_uuid: "f8b13e3a547d4aa4976a23a7ec31e93b" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 35417 } } peers { permanent_uuid: "0047bd8659e148a3926c148821a8b350" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 39919 } }
I20260619 18:51:52.354404 23850 raft_consensus.cc:374] T 97f0c503682846019df0fe41b6baa067 P 0047bd8659e148a3926c148821a8b350 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:51:52.354555 23850 raft_consensus.cc:729] T 97f0c503682846019df0fe41b6baa067 P 0047bd8659e148a3926c148821a8b350 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0047bd8659e148a3926c148821a8b350, State: Initialized, Role: FOLLOWER
I20260619 18:51:52.354960 23849 raft_consensus.cc:348] T 97f0c503682846019df0fe41b6baa067 P f8b13e3a547d4aa4976a23a7ec31e93b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10747c43ea7247b796a05f6e41f21e95" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 34191 } } peers { permanent_uuid: "f8b13e3a547d4aa4976a23a7ec31e93b" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 35417 } } peers { permanent_uuid: "0047bd8659e148a3926c148821a8b350" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 39919 } }
I20260619 18:51:52.355407 23849 raft_consensus.cc:374] T 97f0c503682846019df0fe41b6baa067 P f8b13e3a547d4aa4976a23a7ec31e93b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:51:52.355320 23850 consensus_queue.cc:260] T 97f0c503682846019df0fe41b6baa067 P 0047bd8659e148a3926c148821a8b350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10747c43ea7247b796a05f6e41f21e95" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 34191 } } peers { permanent_uuid: "f8b13e3a547d4aa4976a23a7ec31e93b" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 35417 } } peers { permanent_uuid: "0047bd8659e148a3926c148821a8b350" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 39919 } }
I20260619 18:51:52.355513 23849 raft_consensus.cc:729] T 97f0c503682846019df0fe41b6baa067 P f8b13e3a547d4aa4976a23a7ec31e93b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8b13e3a547d4aa4976a23a7ec31e93b, State: Initialized, Role: FOLLOWER
I20260619 18:51:52.355480 23848 raft_consensus.cc:348] T 97f0c503682846019df0fe41b6baa067 P 10747c43ea7247b796a05f6e41f21e95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10747c43ea7247b796a05f6e41f21e95" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 34191 } } peers { permanent_uuid: "f8b13e3a547d4aa4976a23a7ec31e93b" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 35417 } } peers { permanent_uuid: "0047bd8659e148a3926c148821a8b350" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 39919 } }
I20260619 18:51:52.355930 23848 raft_consensus.cc:374] T 97f0c503682846019df0fe41b6baa067 P 10747c43ea7247b796a05f6e41f21e95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:51:52.356062 23848 raft_consensus.cc:729] T 97f0c503682846019df0fe41b6baa067 P 10747c43ea7247b796a05f6e41f21e95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10747c43ea7247b796a05f6e41f21e95, State: Initialized, Role: FOLLOWER
I20260619 18:51:52.356362 23697 heartbeater.cc:503] Master 127.3.243.126:34521 was elected leader, sending a full tablet report...
I20260619 18:51:52.356241 23849 consensus_queue.cc:260] T 97f0c503682846019df0fe41b6baa067 P f8b13e3a547d4aa4976a23a7ec31e93b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10747c43ea7247b796a05f6e41f21e95" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 34191 } } peers { permanent_uuid: "f8b13e3a547d4aa4976a23a7ec31e93b" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 35417 } } peers { permanent_uuid: "0047bd8659e148a3926c148821a8b350" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 39919 } }
I20260619 18:51:52.357000 23850 ts_tablet_manager.cc:1428] T 97f0c503682846019df0fe41b6baa067 P 0047bd8659e148a3926c148821a8b350: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20260619 18:51:52.356863 23848 consensus_queue.cc:260] T 97f0c503682846019df0fe41b6baa067 P 10747c43ea7247b796a05f6e41f21e95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10747c43ea7247b796a05f6e41f21e95" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 34191 } } peers { permanent_uuid: "f8b13e3a547d4aa4976a23a7ec31e93b" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 35417 } } peers { permanent_uuid: "0047bd8659e148a3926c148821a8b350" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 39919 } }
I20260619 18:51:52.358693 23848 ts_tablet_manager.cc:1428] T 97f0c503682846019df0fe41b6baa067 P 10747c43ea7247b796a05f6e41f21e95: Time spent starting tablet: real 0.013s user 0.011s sys 0.001s
I20260619 18:51:52.359591 23830 heartbeater.cc:503] Master 127.3.243.126:34521 was elected leader, sending a full tablet report...
I20260619 18:51:52.360718 23849 ts_tablet_manager.cc:1428] T 97f0c503682846019df0fe41b6baa067 P f8b13e3a547d4aa4976a23a7ec31e93b: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
I20260619 18:51:52.451284 23856 raft_consensus.cc:482] T 97f0c503682846019df0fe41b6baa067 P 10747c43ea7247b796a05f6e41f21e95 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:51:52.451586 23856 raft_consensus.cc:504] T 97f0c503682846019df0fe41b6baa067 P 10747c43ea7247b796a05f6e41f21e95 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10747c43ea7247b796a05f6e41f21e95" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 34191 } } peers { permanent_uuid: "f8b13e3a547d4aa4976a23a7ec31e93b" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 35417 } } peers { permanent_uuid: "0047bd8659e148a3926c148821a8b350" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 39919 } }
I20260619 18:51:52.453670 23856 leader_election.cc:288] T 97f0c503682846019df0fe41b6baa067 P 10747c43ea7247b796a05f6e41f21e95 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8b13e3a547d4aa4976a23a7ec31e93b (127.3.243.67:35417), 0047bd8659e148a3926c148821a8b350 (127.3.243.66:39919)
I20260619 18:51:52.462299 23845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:52.454098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.65:57793 (local address 127.3.243.67:35417)
0619 18:51:52.454350 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0619 18:51:52.454356 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:51:52.454777 (+ 421us) server_negotiation.cc:407] Connection header received
0619 18:51:52.455088 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:51:52.455094 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:51:52.455167 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:51:52.455226 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:51:52.456111 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:52.457280 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:52.459847 (+ 2567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:52.461267 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:52.461444 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:52.461523 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 18:51:52.461901 (+ 378us) server_negotiation.cc:299] Negotiation successful
0619 18:51:52.462040 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260619 18:51:52.462428 23857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:52.454052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.67:35417 (local address 127.3.243.65:57793)
0619 18:51:52.454681 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:52.454699 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 18:51:52.454905 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:51:52.455455 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:51:52.455459 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:51:52.455487 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:51:52.455919 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:51:52.455935 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:51:52.457417 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:51:52.457423 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:51:52.459662 (+ 2239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:51:52.459678 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:51:52.461496 (+ 1818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:51:52.461502 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:51:52.461720 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:52.461757 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 18:51:52.461873 (+ 116us) client_negotiation.cc:240] Negotiation successful
0619 18:51:52.462013 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"mutex_wait_us":21,"spinlock_wait_cycles":52352,"thread_start_us":304,"threads_started":1}
I20260619 18:51:52.463531 23785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97f0c503682846019df0fe41b6baa067" candidate_uuid: "10747c43ea7247b796a05f6e41f21e95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8b13e3a547d4aa4976a23a7ec31e93b" is_pre_election: true
I20260619 18:51:52.464095 23846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:52.455087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.65:47549 (local address 127.3.243.66:39919)
0619 18:51:52.455323 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0619 18:51:52.455328 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:51:52.455639 (+ 311us) server_negotiation.cc:407] Connection header received
0619 18:51:52.455904 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:51:52.455909 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:51:52.455988 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:51:52.457526 (+ 1538us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:51:52.458210 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:52.459203 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:52.461824 (+ 2621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:52.462964 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:52.463128 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:52.463194 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 18:51:52.463729 (+ 535us) server_negotiation.cc:299] Negotiation successful
0619 18:51:52.463871 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260619 18:51:52.464114 23785 raft_consensus.cc:2454] T 97f0c503682846019df0fe41b6baa067 P f8b13e3a547d4aa4976a23a7ec31e93b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10747c43ea7247b796a05f6e41f21e95 in term 0.
I20260619 18:51:52.464869 23858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:52.454833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.66:39919 (local address 127.3.243.65:47549)
0619 18:51:52.455538 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:52.455554 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 18:51:52.455704 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:51:52.457542 (+ 1838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:51:52.457547 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:51:52.457582 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:51:52.458038 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:51:52.458050 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:51:52.459401 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:51:52.459406 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:51:52.461673 (+ 2267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:51:52.461688 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:51:52.463385 (+ 1697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:51:52.463391 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:51:52.463546 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:52.463579 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 18:51:52.464290 (+ 711us) client_negotiation.cc:240] Negotiation successful
0619 18:51:52.464476 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"mutex_wait_us":2,"spinlock_wait_cycles":896,"thread_start_us":1254,"threads_started":1}
I20260619 18:51:52.464958 23454 leader_election.cc:302] T 97f0c503682846019df0fe41b6baa067 P 10747c43ea7247b796a05f6e41f21e95 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10747c43ea7247b796a05f6e41f21e95, f8b13e3a547d4aa4976a23a7ec31e93b; no voters:
I20260619 18:51:52.465408 23856 raft_consensus.cc:2781] T 97f0c503682846019df0fe41b6baa067 P 10747c43ea7247b796a05f6e41f21e95 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:51:52.465610 23856 raft_consensus.cc:482] T 97f0c503682846019df0fe41b6baa067 P 10747c43ea7247b796a05f6e41f21e95 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:51:52.465764 23856 raft_consensus.cc:3037] T 97f0c503682846019df0fe41b6baa067 P 10747c43ea7247b796a05f6e41f21e95 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:51:52.466233 23652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97f0c503682846019df0fe41b6baa067" candidate_uuid: "10747c43ea7247b796a05f6e41f21e95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0047bd8659e148a3926c148821a8b350" is_pre_election: true
I20260619 18:51:52.466733 23652 raft_consensus.cc:2454] T 97f0c503682846019df0fe41b6baa067 P 0047bd8659e148a3926c148821a8b350 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10747c43ea7247b796a05f6e41f21e95 in term 0.
I20260619 18:51:52.468591 23856 raft_consensus.cc:504] T 97f0c503682846019df0fe41b6baa067 P 10747c43ea7247b796a05f6e41f21e95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10747c43ea7247b796a05f6e41f21e95" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 34191 } } peers { permanent_uuid: "f8b13e3a547d4aa4976a23a7ec31e93b" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 35417 } } peers { permanent_uuid: "0047bd8659e148a3926c148821a8b350" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 39919 } }
I20260619 18:51:52.469199 23856 leader_election.cc:288] T 97f0c503682846019df0fe41b6baa067 P 10747c43ea7247b796a05f6e41f21e95 [CANDIDATE]: Term 1 election: Requested vote from peers f8b13e3a547d4aa4976a23a7ec31e93b (127.3.243.67:35417), 0047bd8659e148a3926c148821a8b350 (127.3.243.66:39919)
I20260619 18:51:52.469655 23652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97f0c503682846019df0fe41b6baa067" candidate_uuid: "10747c43ea7247b796a05f6e41f21e95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0047bd8659e148a3926c148821a8b350"
I20260619 18:51:52.469812 23652 raft_consensus.cc:3037] T 97f0c503682846019df0fe41b6baa067 P 0047bd8659e148a3926c148821a8b350 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:51:52.472455 23652 raft_consensus.cc:2454] T 97f0c503682846019df0fe41b6baa067 P 0047bd8659e148a3926c148821a8b350 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10747c43ea7247b796a05f6e41f21e95 in term 1.
I20260619 18:51:52.472795 23785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97f0c503682846019df0fe41b6baa067" candidate_uuid: "10747c43ea7247b796a05f6e41f21e95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8b13e3a547d4aa4976a23a7ec31e93b"
I20260619 18:51:52.472841 23454 leader_election.cc:302] T 97f0c503682846019df0fe41b6baa067 P 10747c43ea7247b796a05f6e41f21e95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0047bd8659e148a3926c148821a8b350, 10747c43ea7247b796a05f6e41f21e95; no voters:
I20260619 18:51:52.473122 23856 raft_consensus.cc:2781] T 97f0c503682846019df0fe41b6baa067 P 10747c43ea7247b796a05f6e41f21e95 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:51:52.472996 23785 raft_consensus.cc:3037] T 97f0c503682846019df0fe41b6baa067 P f8b13e3a547d4aa4976a23a7ec31e93b [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:51:52.473587 23856 raft_consensus.cc:686] T 97f0c503682846019df0fe41b6baa067 P 10747c43ea7247b796a05f6e41f21e95 [term 1 LEADER]: Becoming Leader. State: Replica: 10747c43ea7247b796a05f6e41f21e95, State: Running, Role: LEADER
I20260619 18:51:52.474124 23856 consensus_queue.cc:237] T 97f0c503682846019df0fe41b6baa067 P 10747c43ea7247b796a05f6e41f21e95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10747c43ea7247b796a05f6e41f21e95" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 34191 } } peers { permanent_uuid: "f8b13e3a547d4aa4976a23a7ec31e93b" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 35417 } } peers { permanent_uuid: "0047bd8659e148a3926c148821a8b350" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 39919 } }
I20260619 18:51:52.476080 23785 raft_consensus.cc:2454] T 97f0c503682846019df0fe41b6baa067 P f8b13e3a547d4aa4976a23a7ec31e93b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10747c43ea7247b796a05f6e41f21e95 in term 1.
I20260619 18:51:52.478716 23372 catalog_manager.cc:5495] T 97f0c503682846019df0fe41b6baa067 P 10747c43ea7247b796a05f6e41f21e95 reported cstate change: term changed from 0 to 1, leader changed from <none> to 10747c43ea7247b796a05f6e41f21e95 (127.3.243.65). New cstate: current_term: 1 leader_uuid: "10747c43ea7247b796a05f6e41f21e95" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10747c43ea7247b796a05f6e41f21e95" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 34191 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8b13e3a547d4aa4976a23a7ec31e93b" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 35417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0047bd8659e148a3926c148821a8b350" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 39919 } health_report { overall_health: UNKNOWN } } }
W20260619 18:51:52.488199 23831 tablet.cc:2369] T 97f0c503682846019df0fe41b6baa067 P f8b13e3a547d4aa4976a23a7ec31e93b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:51:52.515373 23844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:52.510098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53458 (local address 127.3.243.65:34191)
0619 18:51:52.510313 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0619 18:51:52.510319 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:51:52.510338 (+ 19us) server_negotiation.cc:407] Connection header received
0619 18:51:52.510520 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:51:52.510524 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:51:52.510604 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:51:52.510732 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:51:52.511113 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:52.511877 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:52.513292 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:52.513703 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:52.513813 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:52.514332 (+ 519us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:51:52.514815 (+ 483us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:51:52.514960 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 18:51:52.515040 (+ 80us) server_negotiation.cc:299] Negotiation successful
0619 18:51:52.515120 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260619 18:51:52.517407 23698 tablet.cc:2369] T 97f0c503682846019df0fe41b6baa067 P 0047bd8659e148a3926c148821a8b350: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:51:52.523260 23652 raft_consensus.cc:1261] T 97f0c503682846019df0fe41b6baa067 P 0047bd8659e148a3926c148821a8b350 [term 1 FOLLOWER]: Refusing update from remote peer 10747c43ea7247b796a05f6e41f21e95: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:51:52.523283 23785 raft_consensus.cc:1261] T 97f0c503682846019df0fe41b6baa067 P f8b13e3a547d4aa4976a23a7ec31e93b [term 1 FOLLOWER]: Refusing update from remote peer 10747c43ea7247b796a05f6e41f21e95: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:51:52.524094 23859 consensus_queue.cc:1035] T 97f0c503682846019df0fe41b6baa067 P 10747c43ea7247b796a05f6e41f21e95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8b13e3a547d4aa4976a23a7ec31e93b" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 35417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:51:52.524436 23856 consensus_queue.cc:1035] T 97f0c503682846019df0fe41b6baa067 P 10747c43ea7247b796a05f6e41f21e95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0047bd8659e148a3926c148821a8b350" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 39919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:51:52.536314 23862 mvcc.cc:205] Tried to move back new op lower bound from 7298642380886351872 to 7298642380698300416. Current Snapshot: MvccSnapshot[applied={T|T < 7298642380886351872}]
I20260619 18:51:52.548509 23373 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34178:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:51:52.548805 23373 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34178:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:51:52.552417 23373 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0f54603dcc0b449b8a46dc40d1e3e533: Sending DeleteTablet for 3 replicas of tablet 97f0c503682846019df0fe41b6baa067
I20260619 18:51:52.553469 23498 tablet_service.cc:1514] Processing DeleteTablet for tablet 97f0c503682846019df0fe41b6baa067 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:51:52 UTC) from {username='kudu', principal='kudu/127.3.243.126@KRBTEST.COM'} at 127.0.0.1:53454
I20260619 18:51:52.553617 23765 tablet_service.cc:1514] Processing DeleteTablet for tablet 97f0c503682846019df0fe41b6baa067 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:51:52 UTC) from {username='kudu', principal='kudu/127.3.243.126@KRBTEST.COM'} at 127.0.0.1:54700
I20260619 18:51:52.553736 23632 tablet_service.cc:1514] Processing DeleteTablet for tablet 97f0c503682846019df0fe41b6baa067 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:51:52 UTC) from {username='kudu', principal='kudu/127.3.243.126@KRBTEST.COM'} at 127.0.0.1:33936
I20260619 18:51:52.554942 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 23435
I20260619 18:51:52.574223 23872 tablet_replica.cc:330] T 97f0c503682846019df0fe41b6baa067 P 10747c43ea7247b796a05f6e41f21e95: stopping tablet replica
I20260619 18:51:52.574795 23872 raft_consensus.cc:2229] T 97f0c503682846019df0fe41b6baa067 P 10747c43ea7247b796a05f6e41f21e95 [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:51:52.575603 23872 raft_consensus.cc:2258] T 97f0c503682846019df0fe41b6baa067 P 10747c43ea7247b796a05f6e41f21e95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:51:52.576354 23873 tablet_replica.cc:330] T 97f0c503682846019df0fe41b6baa067 P f8b13e3a547d4aa4976a23a7ec31e93b: stopping tablet replica
I20260619 18:51:52.576704 23873 raft_consensus.cc:2229] T 97f0c503682846019df0fe41b6baa067 P f8b13e3a547d4aa4976a23a7ec31e93b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:51:52.577117 23873 raft_consensus.cc:2258] T 97f0c503682846019df0fe41b6baa067 P f8b13e3a547d4aa4976a23a7ec31e93b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:51:52.578078 23872 ts_tablet_manager.cc:1905] T 97f0c503682846019df0fe41b6baa067 P 10747c43ea7247b796a05f6e41f21e95: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 19 18:51:52 dist-test-slave-m9wg krb5kdc[23329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895110, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.65@KRBTEST.COM
I20260619 18:51:52.582398 23873 ts_tablet_manager.cc:1905] T 97f0c503682846019df0fe41b6baa067 P f8b13e3a547d4aa4976a23a7ec31e93b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:51:52.586005 23872 ts_tablet_manager.cc:1918] T 97f0c503682846019df0fe41b6baa067 P 10747c43ea7247b796a05f6e41f21e95: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:51:52.586186 23872 log.cc:1196] T 97f0c503682846019df0fe41b6baa067 P 10747c43ea7247b796a05f6e41f21e95: Deleting WAL directory at /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894810715282-4045-0/minicluster-data/ts-0/wal/wals/97f0c503682846019df0fe41b6baa067
I20260619 18:51:52.586678 23872 ts_tablet_manager.cc:1939] T 97f0c503682846019df0fe41b6baa067 P 10747c43ea7247b796a05f6e41f21e95: Deleting consensus metadata
I20260619 18:51:52.588637 23874 tablet_replica.cc:330] T 97f0c503682846019df0fe41b6baa067 P 0047bd8659e148a3926c148821a8b350: stopping tablet replica
I20260619 18:51:52.589057 23874 raft_consensus.cc:2229] T 97f0c503682846019df0fe41b6baa067 P 0047bd8659e148a3926c148821a8b350 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:51:52.589277 23873 ts_tablet_manager.cc:1918] T 97f0c503682846019df0fe41b6baa067 P f8b13e3a547d4aa4976a23a7ec31e93b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:51:52.589439 23873 log.cc:1196] T 97f0c503682846019df0fe41b6baa067 P f8b13e3a547d4aa4976a23a7ec31e93b: Deleting WAL directory at /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894810715282-4045-0/minicluster-data/ts-2/wal/wals/97f0c503682846019df0fe41b6baa067
I20260619 18:51:52.589581 23874 raft_consensus.cc:2258] T 97f0c503682846019df0fe41b6baa067 P 0047bd8659e148a3926c148821a8b350 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:51:52.589972 23873 ts_tablet_manager.cc:1939] T 97f0c503682846019df0fe41b6baa067 P f8b13e3a547d4aa4976a23a7ec31e93b: Deleting consensus metadata
I20260619 18:51:52.591141 23844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:52.556803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53462 (local address 127.3.243.65:34191)
0619 18:51:52.556989 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 18:51:52.556995 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:51:52.567486 (+ 10491us) server_negotiation.cc:407] Connection header received
0619 18:51:52.568017 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:51:52.568024 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:51:52.568121 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:51:52.568239 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:51:52.570208 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:52.571529 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:52.577481 (+ 5952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:52.577969 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:52.578050 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:52.583028 (+ 4978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:51:52.583047 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:51:52.583067 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:51:52.583106 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:51:52.586974 (+ 3868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:51:52.587980 (+ 1006us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:51:52.587989 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:51:52.587992 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:51:52.588211 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:51:52.588639 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:51:52.588644 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:51:52.588648 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:51:52.589158 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:51:52.589619 (+ 461us) server_negotiation.cc:1035] Waiting for connection context
0619 18:51:52.590554 (+ 935us) server_negotiation.cc:299] Negotiation successful
0619 18:51:52.590801 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 18:51:52.591651 23874 ts_tablet_manager.cc:1905] T 97f0c503682846019df0fe41b6baa067 P 0047bd8659e148a3926c148821a8b350: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:51:52.591759 23361 catalog_manager.cc:4841] TS f8b13e3a547d4aa4976a23a7ec31e93b (127.3.243.67:35417): tablet 97f0c503682846019df0fe41b6baa067 (table test-table [id=03520df56e4f4378b2f7fcb9ca148d0e]) successfully deleted
I20260619 18:51:52.592798 23359 catalog_manager.cc:4841] TS 10747c43ea7247b796a05f6e41f21e95 (127.3.243.65:34191): tablet 97f0c503682846019df0fe41b6baa067 (table test-table [id=03520df56e4f4378b2f7fcb9ca148d0e]) successfully deleted
I20260619 18:51:52.592931 23558 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:51:52.597806 23874 ts_tablet_manager.cc:1918] T 97f0c503682846019df0fe41b6baa067 P 0047bd8659e148a3926c148821a8b350: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:51:52.597929 23874 log.cc:1196] T 97f0c503682846019df0fe41b6baa067 P 0047bd8659e148a3926c148821a8b350: Deleting WAL directory at /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894810715282-4045-0/minicluster-data/ts-1/wal/wals/97f0c503682846019df0fe41b6baa067
I20260619 18:51:52.598321 23874 ts_tablet_manager.cc:1939] T 97f0c503682846019df0fe41b6baa067 P 0047bd8659e148a3926c148821a8b350: Deleting consensus metadata
I20260619 18:51:52.599684 23361 catalog_manager.cc:4841] TS 0047bd8659e148a3926c148821a8b350 (127.3.243.66:39919): tablet 97f0c503682846019df0fe41b6baa067 (table test-table [id=03520df56e4f4378b2f7fcb9ca148d0e]) successfully deleted
I20260619 18:51:52.694029 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 23435
I20260619 18:51:52.713286 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 23569
Jun 19 18:51:52 dist-test-slave-m9wg krb5kdc[23329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895110, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.66@KRBTEST.COM
I20260619 18:51:52.727613 23846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:52.713976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33948 (local address 127.3.243.66:39919)
0619 18:51:52.714169 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 18:51:52.714174 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:51:52.714194 (+ 20us) server_negotiation.cc:407] Connection header received
0619 18:51:52.714377 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:51:52.714382 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:51:52.714453 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:51:52.714535 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:51:52.715769 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:52.716543 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:52.717971 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:52.718358 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:52.718468 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:52.722552 (+ 4084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:51:52.722565 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:51:52.722581 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:51:52.722616 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:51:52.725331 (+ 2715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:51:52.725940 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:51:52.725946 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:51:52.725949 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:51:52.726067 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:51:52.726375 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:51:52.726377 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:51:52.726380 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:51:52.726739 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:51:52.726952 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0619 18:51:52.727223 (+ 271us) server_negotiation.cc:299] Negotiation successful
0619 18:51:52.727374 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 18:51:52.728293 23692 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:51:52.815253 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 23569
I20260619 18:51:52.829015 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 23702
Jun 19 18:51:52 dist-test-slave-m9wg krb5kdc[23329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895110, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.67@KRBTEST.COM
I20260619 18:51:52.843245 23845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:52.829706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54714 (local address 127.3.243.67:35417)
0619 18:51:52.829853 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 18:51:52.829858 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:51:52.829875 (+ 17us) server_negotiation.cc:407] Connection header received
0619 18:51:52.830007 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:51:52.830011 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:51:52.830072 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:51:52.830160 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:51:52.831421 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:52.832178 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:52.833365 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:52.833793 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:52.833890 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:52.838094 (+ 4204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:51:52.838109 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:51:52.838124 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:51:52.838167 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:51:52.840711 (+ 2544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:51:52.841336 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:51:52.841343 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:51:52.841345 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:51:52.841476 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:51:52.841827 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:51:52.841831 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:51:52.841833 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:51:52.842172 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:51:52.842422 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0619 18:51:52.842726 (+ 304us) server_negotiation.cc:299] Negotiation successful
0619 18:51:52.842936 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 18:51:52.843824 23825 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:51:52.929973 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 23702
I20260619 18:51:52.943418 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 23342
I20260619 18:51:52.944492 23404 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:51:53.052946 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 23342
2026-06-19T18:51:53Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (3994 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-19T18:51:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:51:53Z Disabled control of system clock
I20260619 18:51:53.121105 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894810715282-4045-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894810715282-4045-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894810715282-4045-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.126:33641
--webserver_interface=127.3.243.126
--webserver_port=0
--builtin_ntp_servers=127.3.243.84:38793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.126:33641
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 18:51:53.430186 23883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:51:53.430455 23883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:51:53.430534 23883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:51:53.437915 23883 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:51:53.438025 23883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:51:53.438081 23883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:51:53.438136 23883 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:51:53.438179 23883 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:51:53.447258 23883 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:38793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894810715282-4045-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894810715282-4045-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.243.126:33641
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.126:33641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.243.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894810715282-4045-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:51:53.448529 23883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:51:53.450254 23883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:51:53.461040 23888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:51:53.461023 23889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:51:53.461010 23891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:51:53.461930 23883 server_base.cc:1034] running on GCE node
I20260619 18:51:53.462821 23883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:51:53.464114 23883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:51:53.465372 23883 hybrid_clock.cc:648] HybridClock initialized: now 1781895113465320 us; error 43 us; skew 500 ppm
I20260619 18:51:53.465705 23883 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 18:51:53.467844 23883 webserver.cc:460] Webserver started at http://127.3.243.126:38035/ using document root <none> and password file <none>
I20260619 18:51:53.468542 23883 fs_manager.cc:359] Metadata directory not provided
I20260619 18:51:53.468640 23883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:51:53.468930 23883 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:51:53.471669 23883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894810715282-4045-0/minicluster-data/master-0/data/instance:
uuid: "532a05d0408e41b0b9eb76b923f68d2a"
format_stamp: "Formatted at 2026-06-19 18:51:53 on dist-test-slave-m9wg"
server_key: "19ae6ab41a4c5961f82f9f7c01c1262a"
server_key_iv: "5563e61e3a98371c56138541294adb2a"
server_key_version: "encryptionkey@0"
I20260619 18:51:53.472445 23883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894810715282-4045-0/minicluster-data/master-0/wal/instance:
uuid: "532a05d0408e41b0b9eb76b923f68d2a"
format_stamp: "Formatted at 2026-06-19 18:51:53 on dist-test-slave-m9wg"
server_key: "19ae6ab41a4c5961f82f9f7c01c1262a"
server_key_iv: "5563e61e3a98371c56138541294adb2a"
server_key_version: "encryptionkey@0"
I20260619 18:51:53.477200 23883 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 18:51:53.480392 23897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:51:53.481642 23883 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 18:51:53.481796 23883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894810715282-4045-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894810715282-4045-0/minicluster-data/master-0/wal
uuid: "532a05d0408e41b0b9eb76b923f68d2a"
format_stamp: "Formatted at 2026-06-19 18:51:53 on dist-test-slave-m9wg"
server_key: "19ae6ab41a4c5961f82f9f7c01c1262a"
server_key_iv: "5563e61e3a98371c56138541294adb2a"
server_key_version: "encryptionkey@0"
I20260619 18:51:53.481976 23883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894810715282-4045-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894810715282-4045-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894810715282-4045-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:51:53.501293 23883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:51:53.502022 23883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:51:53.502274 23883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:51:53.519316 23883 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.126:33641
I20260619 18:51:53.519338 23948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.126:33641 every 8 connection(s)
I20260619 18:51:53.520610 23883 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
I20260619 18:51:53.524171 23949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:51:53.529273 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 23883
I20260619 18:51:53.529403 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894810715282-4045-0/minicluster-data/master-0/wal/instance
I20260619 18:51:53.529796 4045 external_mini_cluster.cc:1428] Setting key 3384409e3066734bd205b5562beb0c00
I20260619 18:51:53.534531 23949 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 532a05d0408e41b0b9eb76b923f68d2a: Bootstrap starting.
I20260619 18:51:53.538012 23949 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 532a05d0408e41b0b9eb76b923f68d2a: Neither blocks nor log segments found. Creating new log.
I20260619 18:51:53.539237 23949 log.cc:824] T 00000000000000000000000000000000 P 532a05d0408e41b0b9eb76b923f68d2a: Log is configured to *not* fsync() on all Append() calls
I20260619 18:51:53.542284 23949 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 532a05d0408e41b0b9eb76b923f68d2a: No bootstrap required, opened a new log
I20260619 18:51:53.543195 23952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:53.531626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49790 (local address 127.3.243.126:33641)
0619 18:51:53.532625 (+ 999us) server_negotiation.cc:206] Beginning negotiation
0619 18:51:53.532640 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 18:51:53.532679 (+ 39us) server_negotiation.cc:407] Connection header received
0619 18:51:53.534328 (+ 1649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:51:53.534352 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:51:53.534813 (+ 461us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:51:53.535123 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:51:53.535749 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:53.537145 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:53.538997 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:53.539704 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:53.539894 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:53.540959 (+ 1065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:51:53.540967 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:51:53.540980 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:51:53.541514 (+ 534us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:51:53.541590 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:51:53.541611 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:51:53.541698 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 18:51:53.541958 (+ 260us) server_negotiation.cc:299] Negotiation successful
0619 18:51:53.542276 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":358,"threads_started":1}
I20260619 18:51:53.549873 23949 raft_consensus.cc:348] T 00000000000000000000000000000000 P 532a05d0408e41b0b9eb76b923f68d2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "532a05d0408e41b0b9eb76b923f68d2a" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 33641 } }
I20260619 18:51:53.550312 23949 raft_consensus.cc:374] T 00000000000000000000000000000000 P 532a05d0408e41b0b9eb76b923f68d2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:51:53.550417 23949 raft_consensus.cc:729] T 00000000000000000000000000000000 P 532a05d0408e41b0b9eb76b923f68d2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 532a05d0408e41b0b9eb76b923f68d2a, State: Initialized, Role: FOLLOWER
I20260619 18:51:53.550952 23949 consensus_queue.cc:260] T 00000000000000000000000000000000 P 532a05d0408e41b0b9eb76b923f68d2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "532a05d0408e41b0b9eb76b923f68d2a" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 33641 } }
I20260619 18:51:53.551162 23949 raft_consensus.cc:388] T 00000000000000000000000000000000 P 532a05d0408e41b0b9eb76b923f68d2a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:51:53.551285 23949 raft_consensus.cc:482] T 00000000000000000000000000000000 P 532a05d0408e41b0b9eb76b923f68d2a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:51:53.551429 23949 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 532a05d0408e41b0b9eb76b923f68d2a [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:51:53.553625 23949 raft_consensus.cc:504] T 00000000000000000000000000000000 P 532a05d0408e41b0b9eb76b923f68d2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "532a05d0408e41b0b9eb76b923f68d2a" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 33641 } }
I20260619 18:51:53.554103 23949 leader_election.cc:302] T 00000000000000000000000000000000 P 532a05d0408e41b0b9eb76b923f68d2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 532a05d0408e41b0b9eb76b923f68d2a; no voters:
I20260619 18:51:53.554648 23949 leader_election.cc:288] T 00000000000000000000000000000000 P 532a05d0408e41b0b9eb76b923f68d2a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:51:53.554797 23954 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 532a05d0408e41b0b9eb76b923f68d2a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:51:53.555773 23954 raft_consensus.cc:686] T 00000000000000000000000000000000 P 532a05d0408e41b0b9eb76b923f68d2a [term 1 LEADER]: Becoming Leader. State: Replica: 532a05d0408e41b0b9eb76b923f68d2a, State: Running, Role: LEADER
I20260619 18:51:53.556263 23954 consensus_queue.cc:237] T 00000000000000000000000000000000 P 532a05d0408e41b0b9eb76b923f68d2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "532a05d0408e41b0b9eb76b923f68d2a" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 33641 } }
I20260619 18:51:53.556816 23949 sys_catalog.cc:564] T 00000000000000000000000000000000 P 532a05d0408e41b0b9eb76b923f68d2a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:51:53.558779 23955 sys_catalog.cc:455] T 00000000000000000000000000000000 P 532a05d0408e41b0b9eb76b923f68d2a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "532a05d0408e41b0b9eb76b923f68d2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "532a05d0408e41b0b9eb76b923f68d2a" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 33641 } } }
I20260619 18:51:53.559176 23955 sys_catalog.cc:458] T 00000000000000000000000000000000 P 532a05d0408e41b0b9eb76b923f68d2a [sys.catalog]: This master's current role is: LEADER
I20260619 18:51:53.560014 23956 sys_catalog.cc:455] T 00000000000000000000000000000000 P 532a05d0408e41b0b9eb76b923f68d2a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 532a05d0408e41b0b9eb76b923f68d2a. Latest consensus state: current_term: 1 leader_uuid: "532a05d0408e41b0b9eb76b923f68d2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "532a05d0408e41b0b9eb76b923f68d2a" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 33641 } } }
I20260619 18:51:53.560340 23956 sys_catalog.cc:458] T 00000000000000000000000000000000 P 532a05d0408e41b0b9eb76b923f68d2a [sys.catalog]: This master's current role is: LEADER
I20260619 18:51:53.561213 23963 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:51:53.566265 23963 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:51:53.574431 23963 catalog_manager.cc:1347] Generated new cluster ID: c2b40c789b0946128834885fbed6ddf4
I20260619 18:51:53.574528 23963 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:51:53.595647 23963 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:51:53.596939 23963 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:51:53.604018 23963 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 532a05d0408e41b0b9eb76b923f68d2a: Generated new TSK 0
I20260619 18:51:53.604777 23963 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:51:53.612219 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894810715282-4045-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894810715282-4045-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894810715282-4045-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894810715282-4045-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.65:0
--local_ip_for_outbound_sockets=127.3.243.65
--webserver_interface=127.3.243.65
--webserver_port=0
--tserver_master_addrs=127.3.243.126:33641
--builtin_ntp_servers=127.3.243.84:38793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 18:51:53.962603 23973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:51:53.962878 23973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:51:53.962965 23973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:51:53.970499 23973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:51:53.970610 23973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:51:53.970716 23973 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.65
I20260619 18:51:53.979979 23973 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:38793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894810715282-4045-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894810715282-4045-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894810715282-4045-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.243.65
--webserver_port=0
--tserver_master_addrs=127.3.243.126:33641
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.65
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894810715282-4045-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:51:53.981290 23973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:51:53.983060 23973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:51:53.996107 23978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:51:53.996014 23979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:51:53.996048 23981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:51:53.996881 23973 server_base.cc:1034] running on GCE node
I20260619 18:51:53.997531 23973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:51:53.998478 23973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:51:53.999683 23973 hybrid_clock.cc:648] HybridClock initialized: now 1781895113999628 us; error 49 us; skew 500 ppm
I20260619 18:51:54.000005 23973 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 18:51:54.002355 23973 webserver.cc:460] Webserver started at http://127.3.243.65:41943/ using document root <none> and password file <none>
I20260619 18:51:54.003042 23973 fs_manager.cc:359] Metadata directory not provided
I20260619 18:51:54.003155 23973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:51:54.003463 23973 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:51:54.006389 23973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894810715282-4045-0/minicluster-data/ts-0/data/instance:
uuid: "91d0b45aa09e4a418bc838d6ad0414b0"
format_stamp: "Formatted at 2026-06-19 18:51:53 on dist-test-slave-m9wg"
server_key: "505e262892267d9b43a117bff13f48dc"
server_key_iv: "68b5ed4022e46ce22305fe38a6efa358"
server_key_version: "encryptionkey@0"
I20260619 18:51:54.007164 23973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894810715282-4045-0/minicluster-data/ts-0/wal/instance:
uuid: "91d0b45aa09e4a418bc838d6ad0414b0"
format_stamp: "Formatted at 2026-06-19 18:51:53 on dist-test-slave-m9wg"
server_key: "505e262892267d9b43a117bff13f48dc"
server_key_iv: "68b5ed4022e46ce22305fe38a6efa358"
server_key_version: "encryptionkey@0"
I20260619 18:51:54.012207 23973 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260619 18:51:54.015609 23987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:51:54.016954 23973 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:51:54.017122 23973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894810715282-4045-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894810715282-4045-0/minicluster-data/ts-0/wal
uuid: "91d0b45aa09e4a418bc838d6ad0414b0"
format_stamp: "Formatted at 2026-06-19 18:51:53 on dist-test-slave-m9wg"
server_key: "505e262892267d9b43a117bff13f48dc"
server_key_iv: "68b5ed4022e46ce22305fe38a6efa358"
server_key_version: "encryptionkey@0"
I20260619 18:51:54.017320 23973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894810715282-4045-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894810715282-4045-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894810715282-4045-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:51:54.038655 23973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:51:54.039532 23973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:51:54.039819 23973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:51:54.040835 23973 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:51:54.042273 23973 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:51:54.042366 23973 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:51:54.042465 23973 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:51:54.042531 23973 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:51:54.074301 23973 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.65:34057
I20260619 18:51:54.074361 24099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.65:34057 every 8 connection(s)
I20260619 18:51:54.075548 23973 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894810715282-4045-0/minicluster-data/ts-0/data/info.pb
I20260619 18:51:54.080266 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 23973
I20260619 18:51:54.080420 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894810715282-4045-0/minicluster-data/ts-0/wal/instance
I20260619 18:51:54.080842 4045 external_mini_cluster.cc:1428] Setting key 7a740c02b80c57b1698b3d95db1562f6
I20260619 18:51:54.083763 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894810715282-4045-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894810715282-4045-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894810715282-4045-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894810715282-4045-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.66:0
--local_ip_for_outbound_sockets=127.3.243.66
--webserver_interface=127.3.243.66
--webserver_port=0
--tserver_master_addrs=127.3.243.126:33641
--builtin_ntp_servers=127.3.243.84:38793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 18:51:54.088409 24102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:54.077968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.65:35277 (local address 127.3.243.126:33641)
0619 18:51:54.078425 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0619 18:51:54.078432 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:51:54.079605 (+ 1173us) server_negotiation.cc:407] Connection header received
0619 18:51:54.081550 (+ 1945us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:51:54.081555 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:51:54.081644 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:51:54.081755 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:51:54.083125 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:54.084186 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:54.085782 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:54.086312 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:54.086420 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:54.087622 (+ 1202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:51:54.087627 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:51:54.087632 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:51:54.087653 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:51:54.087700 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:51:54.087712 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:51:54.087786 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 18:51:54.087955 (+ 169us) server_negotiation.cc:299] Negotiation successful
0619 18:51:54.088048 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":214,"threads_started":1}
I20260619 18:51:54.089334 24103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:54.078561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:33641 (local address 127.3.243.65:35277)
0619 18:51:54.079456 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:54.079498 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 18:51:54.081225 (+ 1727us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:51:54.082030 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:51:54.082041 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:51:54.082199 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:51:54.082957 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:51:54.082977 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:51:54.084324 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:51:54.084328 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:51:54.085633 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:51:54.085648 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:51:54.086501 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:51:54.086507 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:51:54.086706 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:54.087334 (+ 628us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:51:54.087336 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:51:54.087376 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:51:54.087457 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:51:54.087795 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:51:54.087801 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:51:54.087806 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 18:51:54.087932 (+ 126us) client_negotiation.cc:240] Negotiation successful
0619 18:51:54.088286 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":271,"threads_started":1}
I20260619 18:51:54.090899 24100 heartbeater.cc:347] Connected to a master server at 127.3.243.126:33641
I20260619 18:51:54.091215 24100 heartbeater.cc:464] Registering TS with master...
I20260619 18:51:54.092042 24100 heartbeater.cc:511] Master 127.3.243.126:33641 requested a full tablet report, sending...
I20260619 18:51:54.094013 23914 ts_manager.cc:194] Registered new tserver with Master: 91d0b45aa09e4a418bc838d6ad0414b0 (127.3.243.65:34057)
I20260619 18:51:54.095485 23914 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.243.65:35277
W20260619 18:51:54.398870 24104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:51:54.399147 24104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:51:54.399266 24104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:51:54.406479 24104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:51:54.406589 24104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:51:54.406731 24104 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.66
I20260619 18:51:54.416329 24104 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:38793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894810715282-4045-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894810715282-4045-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894810715282-4045-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.243.66
--webserver_port=0
--tserver_master_addrs=127.3.243.126:33641
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.66
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894810715282-4045-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:51:54.417634 24104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:51:54.419415 24104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:51:54.431788 24109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:51:54.431901 24112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:51:54.431993 24110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:51:54.432853 24104 server_base.cc:1034] running on GCE node
I20260619 18:51:54.433447 24104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:51:54.434326 24104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:51:54.435527 24104 hybrid_clock.cc:648] HybridClock initialized: now 1781895114435477 us; error 46 us; skew 500 ppm
I20260619 18:51:54.435822 24104 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 18:51:54.437952 24104 webserver.cc:460] Webserver started at http://127.3.243.66:40327/ using document root <none> and password file <none>
I20260619 18:51:54.438596 24104 fs_manager.cc:359] Metadata directory not provided
I20260619 18:51:54.438691 24104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:51:54.438985 24104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:51:54.441982 24104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894810715282-4045-0/minicluster-data/ts-1/data/instance:
uuid: "49d19883f290401f82b63c29f0c1b3a8"
format_stamp: "Formatted at 2026-06-19 18:51:54 on dist-test-slave-m9wg"
server_key: "af9d14d86b95aa3e27250179260359e6"
server_key_iv: "bcf1fb43004faf251d4bb58fd754a923"
server_key_version: "encryptionkey@0"
I20260619 18:51:54.442767 24104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894810715282-4045-0/minicluster-data/ts-1/wal/instance:
uuid: "49d19883f290401f82b63c29f0c1b3a8"
format_stamp: "Formatted at 2026-06-19 18:51:54 on dist-test-slave-m9wg"
server_key: "af9d14d86b95aa3e27250179260359e6"
server_key_iv: "bcf1fb43004faf251d4bb58fd754a923"
server_key_version: "encryptionkey@0"
I20260619 18:51:54.448278 24104 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260619 18:51:54.451874 24118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:51:54.453249 24104 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 18:51:54.453410 24104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894810715282-4045-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894810715282-4045-0/minicluster-data/ts-1/wal
uuid: "49d19883f290401f82b63c29f0c1b3a8"
format_stamp: "Formatted at 2026-06-19 18:51:54 on dist-test-slave-m9wg"
server_key: "af9d14d86b95aa3e27250179260359e6"
server_key_iv: "bcf1fb43004faf251d4bb58fd754a923"
server_key_version: "encryptionkey@0"
I20260619 18:51:54.453619 24104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894810715282-4045-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894810715282-4045-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894810715282-4045-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:51:54.490430 24104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:51:54.491317 24104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:51:54.491629 24104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:51:54.492641 24104 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:51:54.494091 24104 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:51:54.494181 24104 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:51:54.494289 24104 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:51:54.494366 24104 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:51:54.524111 24104 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.66:46335
I20260619 18:51:54.524175 24230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.66:46335 every 8 connection(s)
I20260619 18:51:54.525372 24104 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894810715282-4045-0/minicluster-data/ts-1/data/info.pb
I20260619 18:51:54.528740 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 24104
I20260619 18:51:54.528895 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894810715282-4045-0/minicluster-data/ts-1/wal/instance
I20260619 18:51:54.529506 4045 external_mini_cluster.cc:1428] Setting key 85b73ef241bf80140d0f2b530c2973cc
I20260619 18:51:54.532617 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894810715282-4045-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894810715282-4045-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894810715282-4045-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894810715282-4045-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.67:0
--local_ip_for_outbound_sockets=127.3.243.67
--webserver_interface=127.3.243.67
--webserver_port=0
--tserver_master_addrs=127.3.243.126:33641
--builtin_ntp_servers=127.3.243.84:38793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 18:51:54.539189 24102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:54.527914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.66:38891 (local address 127.3.243.126:33641)
0619 18:51:54.528100 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 18:51:54.528104 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:51:54.530306 (+ 2202us) server_negotiation.cc:407] Connection header received
0619 18:51:54.532036 (+ 1730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:51:54.532043 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:51:54.532124 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:51:54.532254 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:51:54.533526 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:54.534652 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:54.536603 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:54.536999 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:54.537099 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:54.538518 (+ 1419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:51:54.538522 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:51:54.538526 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:51:54.538545 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:51:54.538578 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:51:54.538589 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:51:54.538655 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 18:51:54.538860 (+ 205us) server_negotiation.cc:299] Negotiation successful
0619 18:51:54.538942 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 18:51:54.540663 24233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:54.528697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:33641 (local address 127.3.243.66:38891)
0619 18:51:54.529882 (+ 1185us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:54.529928 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 18:51:54.531743 (+ 1815us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:51:54.532483 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:51:54.532495 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:51:54.532624 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:51:54.533358 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:51:54.533381 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:51:54.534800 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:51:54.534805 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:51:54.536474 (+ 1669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:51:54.536488 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:51:54.537206 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:51:54.537212 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:51:54.537463 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:54.538180 (+ 717us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:51:54.538183 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:51:54.538235 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:51:54.538342 (+ 107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:51:54.538668 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:51:54.538676 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:51:54.538684 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 18:51:54.538853 (+ 169us) client_negotiation.cc:240] Negotiation successful
0619 18:51:54.539262 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":306,"threads_started":1}
I20260619 18:51:54.542219 24231 heartbeater.cc:347] Connected to a master server at 127.3.243.126:33641
I20260619 18:51:54.542539 24231 heartbeater.cc:464] Registering TS with master...
I20260619 18:51:54.543344 24231 heartbeater.cc:511] Master 127.3.243.126:33641 requested a full tablet report, sending...
I20260619 18:51:54.544955 23914 ts_manager.cc:194] Registered new tserver with Master: 49d19883f290401f82b63c29f0c1b3a8 (127.3.243.66:46335)
I20260619 18:51:54.546029 23914 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.243.66:38891
W20260619 18:51:54.877571 24234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:51:54.877866 24234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:51:54.877955 24234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:51:54.885213 24234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:51:54.885329 24234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:51:54.885464 24234 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.67
I20260619 18:51:54.894606 24234 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:38793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894810715282-4045-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894810715282-4045-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894810715282-4045-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.243.67
--webserver_port=0
--tserver_master_addrs=127.3.243.126:33641
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.67
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894810715282-4045-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:51:54.895838 24234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:51:54.897539 24234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:51:54.909673 24239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:51:54.909698 24240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:51:54.909917 24242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:51:54.910566 24234 server_base.cc:1034] running on GCE node
I20260619 18:51:54.911134 24234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:51:54.912020 24234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:51:54.913232 24234 hybrid_clock.cc:648] HybridClock initialized: now 1781895114913190 us; error 58 us; skew 500 ppm
I20260619 18:51:54.913527 24234 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 18:51:54.915715 24234 webserver.cc:460] Webserver started at http://127.3.243.67:45501/ using document root <none> and password file <none>
I20260619 18:51:54.916445 24234 fs_manager.cc:359] Metadata directory not provided
I20260619 18:51:54.916545 24234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:51:54.916843 24234 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:51:54.919634 24234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894810715282-4045-0/minicluster-data/ts-2/data/instance:
uuid: "d2e61ba10bc246799d748110a8f4e188"
format_stamp: "Formatted at 2026-06-19 18:51:54 on dist-test-slave-m9wg"
server_key: "fc6fd4ba6344c571033c9c17b959daa7"
server_key_iv: "323ab62699c71d22fd0590517580bc41"
server_key_version: "encryptionkey@0"
I20260619 18:51:54.920449 24234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894810715282-4045-0/minicluster-data/ts-2/wal/instance:
uuid: "d2e61ba10bc246799d748110a8f4e188"
format_stamp: "Formatted at 2026-06-19 18:51:54 on dist-test-slave-m9wg"
server_key: "fc6fd4ba6344c571033c9c17b959daa7"
server_key_iv: "323ab62699c71d22fd0590517580bc41"
server_key_version: "encryptionkey@0"
I20260619 18:51:54.925498 24234 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 18:51:54.928963 24248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:51:54.930370 24234 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 18:51:54.930529 24234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894810715282-4045-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894810715282-4045-0/minicluster-data/ts-2/wal
uuid: "d2e61ba10bc246799d748110a8f4e188"
format_stamp: "Formatted at 2026-06-19 18:51:54 on dist-test-slave-m9wg"
server_key: "fc6fd4ba6344c571033c9c17b959daa7"
server_key_iv: "323ab62699c71d22fd0590517580bc41"
server_key_version: "encryptionkey@0"
I20260619 18:51:54.930737 24234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894810715282-4045-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894810715282-4045-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894810715282-4045-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:51:54.950881 24234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:51:54.951877 24234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:51:54.952224 24234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:51:54.953293 24234 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:51:54.954881 24234 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:51:54.954978 24234 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:51:54.955096 24234 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:51:54.955173 24234 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:51:54.986784 24234 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.67:40897
I20260619 18:51:54.986825 24360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.67:40897 every 8 connection(s)
I20260619 18:51:54.988175 24234 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894810715282-4045-0/minicluster-data/ts-2/data/info.pb
I20260619 18:51:54.991290 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 24234
I20260619 18:51:54.991477 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894810715282-4045-0/minicluster-data/ts-2/wal/instance
I20260619 18:51:54.992038 4045 external_mini_cluster.cc:1428] Setting key d645fe90496eef5b2916b63d9373f08d
I20260619 18:51:55.001531 24102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:54.990829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.67:58783 (local address 127.3.243.126:33641)
0619 18:51:54.990995 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 18:51:54.990999 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:51:54.992951 (+ 1952us) server_negotiation.cc:407] Connection header received
0619 18:51:54.995010 (+ 2059us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:51:54.995016 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:51:54.995088 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:51:54.995167 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:51:54.996224 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:54.996946 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:54.998567 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:54.999023 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:54.999133 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:55.000703 (+ 1570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:51:55.000711 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:51:55.000714 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:51:55.000735 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:51:55.000771 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:51:55.000783 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:51:55.000856 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 18:51:55.001169 (+ 313us) server_negotiation.cc:299] Negotiation successful
0619 18:51:55.001275 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 18:51:55.002609 24363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:54.991483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:33641 (local address 127.3.243.67:58783)
0619 18:51:54.992806 (+ 1323us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:54.992855 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 18:51:54.994713 (+ 1858us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:51:54.995365 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:51:54.995373 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:51:54.995470 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:51:54.996059 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:51:54.996077 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:51:54.997077 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:51:54.997081 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:51:54.998403 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:51:54.998416 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:51:54.999287 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:51:54.999293 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:51:54.999524 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:55.000309 (+ 785us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:51:55.000312 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:51:55.000357 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:51:55.000456 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:51:55.000896 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:51:55.000906 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:51:55.000913 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 18:51:55.001126 (+ 213us) client_negotiation.cc:240] Negotiation successful
0619 18:51:55.001481 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":698,"thread_start_us":335,"threads_started":1}
I20260619 18:51:55.004065 24361 heartbeater.cc:347] Connected to a master server at 127.3.243.126:33641
I20260619 18:51:55.004413 24361 heartbeater.cc:464] Registering TS with master...
I20260619 18:51:55.005220 24361 heartbeater.cc:511] Master 127.3.243.126:33641 requested a full tablet report, sending...
I20260619 18:51:55.006768 23914 ts_manager.cc:194] Registered new tserver with Master: d2e61ba10bc246799d748110a8f4e188 (127.3.243.67:40897)
I20260619 18:51:55.007572 4045 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 18:51:55.007736 23914 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.243.67:58783
I20260619 18:51:55.017302 24102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:55.011565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49800 (local address 127.3.243.126:33641)
0619 18:51:55.011781 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0619 18:51:55.011788 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:51:55.012182 (+ 394us) server_negotiation.cc:407] Connection header received
0619 18:51:55.012367 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:51:55.012371 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:51:55.012444 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:51:55.012525 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:51:55.013010 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:55.013711 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:55.015044 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:55.015436 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:55.015545 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:55.016629 (+ 1084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:51:55.016633 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:51:55.016636 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:51:55.016655 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:51:55.016687 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:51:55.016698 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:51:55.016772 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 18:51:55.016911 (+ 139us) server_negotiation.cc:299] Negotiation successful
0619 18:51:55.017010 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260619 18:51:55.021451 23914 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49800:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:51:55.028529 23914 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:51:55.063859 24373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:55.040568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.66:46335 (local address 127.0.0.1:37570)
0619 18:51:55.054164 (+ 13596us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:55.054229 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0619 18:51:55.054611 (+ 382us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:51:55.055737 (+ 1126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:51:55.055746 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:51:55.055784 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:51:55.056243 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:51:55.056257 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:51:55.058240 (+ 1983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:51:55.058249 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:51:55.060944 (+ 2695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:51:55.060960 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:51:55.063045 (+ 2085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:51:55.063050 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:51:55.063201 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:55.063236 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 18:51:55.063337 (+ 101us) client_negotiation.cc:240] Negotiation successful
0619 18:51:55.063526 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13240,"thread_start_us":13078,"threads_started":1}
I20260619 18:51:55.063913 24375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:55.040912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37570 (local address 127.3.243.66:46335)
0619 18:51:55.053193 (+ 12281us) server_negotiation.cc:206] Beginning negotiation
0619 18:51:55.053204 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 18:51:55.054395 (+ 1191us) server_negotiation.cc:407] Connection header received
0619 18:51:55.054919 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:51:55.054930 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:51:55.055252 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:51:55.055418 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:51:55.056457 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:55.058060 (+ 1603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:55.061149 (+ 3089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:55.062522 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:55.062656 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:55.062941 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0619 18:51:55.063437 (+ 496us) server_negotiation.cc:299] Negotiation successful
0619 18:51:55.063623 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12045,"thread_start_us":12306,"threads_started":1}
I20260619 18:51:55.068356 24166 tablet_service.cc:1467] Processing CreateTablet for tablet 6e1495389b1d4657826ca244d93f6626 (DEFAULT_TABLE table=test-table [id=9baf262c37d04b4abf95b0fdc4c4adb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:51:55.070053 24166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e1495389b1d4657826ca244d93f6626. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:51:55.070762 24376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:55.041296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.67:40897 (local address 127.0.0.1:38408)
0619 18:51:55.060892 (+ 19596us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:55.060911 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 18:51:55.061035 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:51:55.061785 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:51:55.061789 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:51:55.061825 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:51:55.062681 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:51:55.062726 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:51:55.064696 (+ 1970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:51:55.064701 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:51:55.067873 (+ 3172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:51:55.067890 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:51:55.069822 (+ 1932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:51:55.069829 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:51:55.070009 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:55.070042 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 18:51:55.070197 (+ 155us) client_negotiation.cc:240] Negotiation successful
0619 18:51:55.070346 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19470,"mutex_wait_us":61,"spinlock_wait_cycles":1408,"thread_start_us":19316,"threads_started":1}
I20260619 18:51:55.071506 24377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:55.041966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38408 (local address 127.3.243.67:40897)
0619 18:51:55.059341 (+ 17375us) server_negotiation.cc:206] Beginning negotiation
0619 18:51:55.059351 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:51:55.061017 (+ 1666us) server_negotiation.cc:407] Connection header received
0619 18:51:55.061152 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:51:55.061158 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:51:55.061327 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:51:55.061526 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:51:55.062922 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:55.064557 (+ 1635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:55.068134 (+ 3577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:55.069611 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:55.069750 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:55.070021 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0619 18:51:55.071003 (+ 982us) server_negotiation.cc:299] Negotiation successful
0619 18:51:55.071186 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17171,"thread_start_us":17392,"threads_started":1}
I20260619 18:51:55.072064 24374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:55.040959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.65:34057 (local address 127.0.0.1:36800)
0619 18:51:55.060559 (+ 19600us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:55.060584 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 18:51:55.060818 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:51:55.061784 (+ 966us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:51:55.061789 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:51:55.061834 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:51:55.062783 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:51:55.062796 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:51:55.064880 (+ 2084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:51:55.064886 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:51:55.069221 (+ 4335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:51:55.069236 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:51:55.071292 (+ 2056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:51:55.071298 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:51:55.071466 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:55.071500 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 18:51:55.071582 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 18:51:55.071712 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19436,"mutex_wait_us":107,"spinlock_wait_cycles":3456,"thread_start_us":19272,"threads_started":1}
I20260619 18:51:55.072750 24378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:55.042041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36800 (local address 127.3.243.65:34057)
0619 18:51:55.060067 (+ 18026us) server_negotiation.cc:206] Beginning negotiation
0619 18:51:55.060077 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:51:55.060706 (+ 629us) server_negotiation.cc:407] Connection header received
0619 18:51:55.061013 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:51:55.061022 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:51:55.061326 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:51:55.061526 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:51:55.063003 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:55.064750 (+ 1747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:55.069422 (+ 4672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:55.070790 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:55.070921 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:55.071169 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0619 18:51:55.072280 (+ 1111us) server_negotiation.cc:299] Negotiation successful
0619 18:51:55.072460 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17851,"thread_start_us":17643,"threads_started":1}
I20260619 18:51:55.075353 24296 tablet_service.cc:1467] Processing CreateTablet for tablet 6e1495389b1d4657826ca244d93f6626 (DEFAULT_TABLE table=test-table [id=9baf262c37d04b4abf95b0fdc4c4adb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:51:55.076819 24296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e1495389b1d4657826ca244d93f6626. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:51:55.080176 24035 tablet_service.cc:1467] Processing CreateTablet for tablet 6e1495389b1d4657826ca244d93f6626 (DEFAULT_TABLE table=test-table [id=9baf262c37d04b4abf95b0fdc4c4adb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:51:55.081668 24035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e1495389b1d4657826ca244d93f6626. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:51:55.089941 24380 tablet_bootstrap.cc:492] T 6e1495389b1d4657826ca244d93f6626 P d2e61ba10bc246799d748110a8f4e188: Bootstrap starting.
I20260619 18:51:55.091174 24379 tablet_bootstrap.cc:492] T 6e1495389b1d4657826ca244d93f6626 P 49d19883f290401f82b63c29f0c1b3a8: Bootstrap starting.
I20260619 18:51:55.093811 24381 tablet_bootstrap.cc:492] T 6e1495389b1d4657826ca244d93f6626 P 91d0b45aa09e4a418bc838d6ad0414b0: Bootstrap starting.
I20260619 18:51:55.094874 24379 tablet_bootstrap.cc:654] T 6e1495389b1d4657826ca244d93f6626 P 49d19883f290401f82b63c29f0c1b3a8: Neither blocks nor log segments found. Creating new log.
I20260619 18:51:55.094879 24380 tablet_bootstrap.cc:654] T 6e1495389b1d4657826ca244d93f6626 P d2e61ba10bc246799d748110a8f4e188: Neither blocks nor log segments found. Creating new log.
I20260619 18:51:55.095888 24379 log.cc:824] T 6e1495389b1d4657826ca244d93f6626 P 49d19883f290401f82b63c29f0c1b3a8: Log is configured to *not* fsync() on all Append() calls
I20260619 18:51:55.096102 24380 log.cc:824] T 6e1495389b1d4657826ca244d93f6626 P d2e61ba10bc246799d748110a8f4e188: Log is configured to *not* fsync() on all Append() calls
I20260619 18:51:55.097549 24381 tablet_bootstrap.cc:654] T 6e1495389b1d4657826ca244d93f6626 P 91d0b45aa09e4a418bc838d6ad0414b0: Neither blocks nor log segments found. Creating new log.
I20260619 18:51:55.098284 24100 heartbeater.cc:503] Master 127.3.243.126:33641 was elected leader, sending a full tablet report...
I20260619 18:51:55.098937 24381 log.cc:824] T 6e1495389b1d4657826ca244d93f6626 P 91d0b45aa09e4a418bc838d6ad0414b0: Log is configured to *not* fsync() on all Append() calls
I20260619 18:51:55.102876 24380 tablet_bootstrap.cc:492] T 6e1495389b1d4657826ca244d93f6626 P d2e61ba10bc246799d748110a8f4e188: No bootstrap required, opened a new log
I20260619 18:51:55.102986 24379 tablet_bootstrap.cc:492] T 6e1495389b1d4657826ca244d93f6626 P 49d19883f290401f82b63c29f0c1b3a8: No bootstrap required, opened a new log
I20260619 18:51:55.103185 24380 ts_tablet_manager.cc:1397] T 6e1495389b1d4657826ca244d93f6626 P d2e61ba10bc246799d748110a8f4e188: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.000s
I20260619 18:51:55.103253 24379 ts_tablet_manager.cc:1397] T 6e1495389b1d4657826ca244d93f6626 P 49d19883f290401f82b63c29f0c1b3a8: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.003s
I20260619 18:51:55.110002 24381 tablet_bootstrap.cc:492] T 6e1495389b1d4657826ca244d93f6626 P 91d0b45aa09e4a418bc838d6ad0414b0: No bootstrap required, opened a new log
I20260619 18:51:55.110350 24381 ts_tablet_manager.cc:1397] T 6e1495389b1d4657826ca244d93f6626 P 91d0b45aa09e4a418bc838d6ad0414b0: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.000s
I20260619 18:51:55.111315 24380 raft_consensus.cc:348] T 6e1495389b1d4657826ca244d93f6626 P d2e61ba10bc246799d748110a8f4e188 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49d19883f290401f82b63c29f0c1b3a8" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 46335 } } peers { permanent_uuid: "91d0b45aa09e4a418bc838d6ad0414b0" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 34057 } } peers { permanent_uuid: "d2e61ba10bc246799d748110a8f4e188" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40897 } }
I20260619 18:51:55.111343 24379 raft_consensus.cc:348] T 6e1495389b1d4657826ca244d93f6626 P 49d19883f290401f82b63c29f0c1b3a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49d19883f290401f82b63c29f0c1b3a8" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 46335 } } peers { permanent_uuid: "91d0b45aa09e4a418bc838d6ad0414b0" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 34057 } } peers { permanent_uuid: "d2e61ba10bc246799d748110a8f4e188" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40897 } }
I20260619 18:51:55.111688 24380 raft_consensus.cc:374] T 6e1495389b1d4657826ca244d93f6626 P d2e61ba10bc246799d748110a8f4e188 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:51:55.111784 24379 raft_consensus.cc:374] T 6e1495389b1d4657826ca244d93f6626 P 49d19883f290401f82b63c29f0c1b3a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:51:55.111789 24380 raft_consensus.cc:729] T 6e1495389b1d4657826ca244d93f6626 P d2e61ba10bc246799d748110a8f4e188 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2e61ba10bc246799d748110a8f4e188, State: Initialized, Role: FOLLOWER
I20260619 18:51:55.111894 24379 raft_consensus.cc:729] T 6e1495389b1d4657826ca244d93f6626 P 49d19883f290401f82b63c29f0c1b3a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49d19883f290401f82b63c29f0c1b3a8, State: Initialized, Role: FOLLOWER
I20260619 18:51:55.112435 24380 consensus_queue.cc:260] T 6e1495389b1d4657826ca244d93f6626 P d2e61ba10bc246799d748110a8f4e188 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49d19883f290401f82b63c29f0c1b3a8" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 46335 } } peers { permanent_uuid: "91d0b45aa09e4a418bc838d6ad0414b0" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 34057 } } peers { permanent_uuid: "d2e61ba10bc246799d748110a8f4e188" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40897 } }
I20260619 18:51:55.112572 24379 consensus_queue.cc:260] T 6e1495389b1d4657826ca244d93f6626 P 49d19883f290401f82b63c29f0c1b3a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49d19883f290401f82b63c29f0c1b3a8" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 46335 } } peers { permanent_uuid: "91d0b45aa09e4a418bc838d6ad0414b0" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 34057 } } peers { permanent_uuid: "d2e61ba10bc246799d748110a8f4e188" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40897 } }
I20260619 18:51:55.113495 24361 heartbeater.cc:503] Master 127.3.243.126:33641 was elected leader, sending a full tablet report...
I20260619 18:51:55.114210 24380 ts_tablet_manager.cc:1428] T 6e1495389b1d4657826ca244d93f6626 P d2e61ba10bc246799d748110a8f4e188: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260619 18:51:55.115195 24231 heartbeater.cc:503] Master 127.3.243.126:33641 was elected leader, sending a full tablet report...
I20260619 18:51:55.117065 24379 ts_tablet_manager.cc:1428] T 6e1495389b1d4657826ca244d93f6626 P 49d19883f290401f82b63c29f0c1b3a8: Time spent starting tablet: real 0.014s user 0.012s sys 0.001s
I20260619 18:51:55.120419 24381 raft_consensus.cc:348] T 6e1495389b1d4657826ca244d93f6626 P 91d0b45aa09e4a418bc838d6ad0414b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49d19883f290401f82b63c29f0c1b3a8" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 46335 } } peers { permanent_uuid: "91d0b45aa09e4a418bc838d6ad0414b0" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 34057 } } peers { permanent_uuid: "d2e61ba10bc246799d748110a8f4e188" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40897 } }
I20260619 18:51:55.120839 24381 raft_consensus.cc:374] T 6e1495389b1d4657826ca244d93f6626 P 91d0b45aa09e4a418bc838d6ad0414b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:51:55.120934 24381 raft_consensus.cc:729] T 6e1495389b1d4657826ca244d93f6626 P 91d0b45aa09e4a418bc838d6ad0414b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91d0b45aa09e4a418bc838d6ad0414b0, State: Initialized, Role: FOLLOWER
I20260619 18:51:55.121505 24381 consensus_queue.cc:260] T 6e1495389b1d4657826ca244d93f6626 P 91d0b45aa09e4a418bc838d6ad0414b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49d19883f290401f82b63c29f0c1b3a8" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 46335 } } peers { permanent_uuid: "91d0b45aa09e4a418bc838d6ad0414b0" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 34057 } } peers { permanent_uuid: "d2e61ba10bc246799d748110a8f4e188" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40897 } }
I20260619 18:51:55.122929 24381 ts_tablet_manager.cc:1428] T 6e1495389b1d4657826ca244d93f6626 P 91d0b45aa09e4a418bc838d6ad0414b0: Time spent starting tablet: real 0.012s user 0.014s sys 0.000s
I20260619 18:51:55.150983 24385 raft_consensus.cc:482] T 6e1495389b1d4657826ca244d93f6626 P d2e61ba10bc246799d748110a8f4e188 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:51:55.151322 24385 raft_consensus.cc:504] T 6e1495389b1d4657826ca244d93f6626 P d2e61ba10bc246799d748110a8f4e188 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49d19883f290401f82b63c29f0c1b3a8" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 46335 } } peers { permanent_uuid: "91d0b45aa09e4a418bc838d6ad0414b0" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 34057 } } peers { permanent_uuid: "d2e61ba10bc246799d748110a8f4e188" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40897 } }
I20260619 18:51:55.153151 24385 leader_election.cc:288] T 6e1495389b1d4657826ca244d93f6626 P d2e61ba10bc246799d748110a8f4e188 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49d19883f290401f82b63c29f0c1b3a8 (127.3.243.66:46335), 91d0b45aa09e4a418bc838d6ad0414b0 (127.3.243.65:34057)
I20260619 18:51:55.159914 24363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:55.153585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.65:34057 (local address 127.3.243.67:35353)
0619 18:51:55.153792 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:55.153813 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 18:51:55.153939 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:51:55.154287 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:51:55.154292 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:51:55.154318 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:51:55.154562 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:51:55.154574 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:51:55.155794 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:51:55.155799 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:51:55.157797 (+ 1998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:51:55.157812 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:51:55.159269 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:51:55.159273 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:51:55.159424 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:55.159450 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 18:51:55.159524 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 18:51:55.159646 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260619 18:51:55.159967 24378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:55.153750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.67:35353 (local address 127.3.243.65:34057)
0619 18:51:55.153935 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 18:51:55.153941 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:51:55.153961 (+ 20us) server_negotiation.cc:407] Connection header received
0619 18:51:55.154084 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:51:55.154090 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:51:55.154170 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:51:55.154244 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:51:55.154779 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:55.155660 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:55.157987 (+ 2327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:55.159123 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:55.159277 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:55.159360 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 18:51:55.159564 (+ 204us) server_negotiation.cc:299] Negotiation successful
0619 18:51:55.159698 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 18:51:55.161015 24375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:55.153751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.67:39125 (local address 127.3.243.66:46335)
0619 18:51:55.153912 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 18:51:55.153916 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:51:55.154921 (+ 1005us) server_negotiation.cc:407] Connection header received
0619 18:51:55.155004 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:51:55.155009 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:51:55.155094 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:51:55.155196 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:51:55.155709 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:55.156508 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:55.158613 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:55.159726 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:55.159879 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:55.159941 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 18:51:55.160724 (+ 783us) server_negotiation.cc:299] Negotiation successful
0619 18:51:55.160842 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 18:51:55.161087 24388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:55.154126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.66:46335 (local address 127.3.243.67:39125)
0619 18:51:55.154820 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0619 18:51:55.154837 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 18:51:55.154921 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:51:55.155244 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:51:55.155255 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:51:55.155277 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:51:55.155554 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:51:55.155561 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:51:55.156664 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:51:55.156669 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:51:55.158453 (+ 1784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:51:55.158464 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:51:55.159889 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:51:55.159895 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:51:55.160059 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:55.160092 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 18:51:55.160687 (+ 595us) client_negotiation.cc:240] Negotiation successful
0619 18:51:55.160815 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":383,"threads_started":1}
I20260619 18:51:55.161125 24055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e1495389b1d4657826ca244d93f6626" candidate_uuid: "d2e61ba10bc246799d748110a8f4e188" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91d0b45aa09e4a418bc838d6ad0414b0" is_pre_election: true
I20260619 18:51:55.161692 24055 raft_consensus.cc:2454] T 6e1495389b1d4657826ca244d93f6626 P 91d0b45aa09e4a418bc838d6ad0414b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2e61ba10bc246799d748110a8f4e188 in term 0.
I20260619 18:51:55.162115 24186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e1495389b1d4657826ca244d93f6626" candidate_uuid: "d2e61ba10bc246799d748110a8f4e188" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49d19883f290401f82b63c29f0c1b3a8" is_pre_election: true
I20260619 18:51:55.162441 24252 leader_election.cc:302] T 6e1495389b1d4657826ca244d93f6626 P d2e61ba10bc246799d748110a8f4e188 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91d0b45aa09e4a418bc838d6ad0414b0, d2e61ba10bc246799d748110a8f4e188; no voters:
I20260619 18:51:55.162665 24186 raft_consensus.cc:2454] T 6e1495389b1d4657826ca244d93f6626 P 49d19883f290401f82b63c29f0c1b3a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2e61ba10bc246799d748110a8f4e188 in term 0.
I20260619 18:51:55.162873 24385 raft_consensus.cc:2781] T 6e1495389b1d4657826ca244d93f6626 P d2e61ba10bc246799d748110a8f4e188 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:51:55.163014 24385 raft_consensus.cc:482] T 6e1495389b1d4657826ca244d93f6626 P d2e61ba10bc246799d748110a8f4e188 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:51:55.163127 24385 raft_consensus.cc:3037] T 6e1495389b1d4657826ca244d93f6626 P d2e61ba10bc246799d748110a8f4e188 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:51:55.165484 24385 raft_consensus.cc:504] T 6e1495389b1d4657826ca244d93f6626 P d2e61ba10bc246799d748110a8f4e188 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49d19883f290401f82b63c29f0c1b3a8" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 46335 } } peers { permanent_uuid: "91d0b45aa09e4a418bc838d6ad0414b0" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 34057 } } peers { permanent_uuid: "d2e61ba10bc246799d748110a8f4e188" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40897 } }
I20260619 18:51:55.166062 24385 leader_election.cc:288] T 6e1495389b1d4657826ca244d93f6626 P d2e61ba10bc246799d748110a8f4e188 [CANDIDATE]: Term 1 election: Requested vote from peers 49d19883f290401f82b63c29f0c1b3a8 (127.3.243.66:46335), 91d0b45aa09e4a418bc838d6ad0414b0 (127.3.243.65:34057)
I20260619 18:51:55.166473 24186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e1495389b1d4657826ca244d93f6626" candidate_uuid: "d2e61ba10bc246799d748110a8f4e188" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49d19883f290401f82b63c29f0c1b3a8"
I20260619 18:51:55.166571 24055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e1495389b1d4657826ca244d93f6626" candidate_uuid: "d2e61ba10bc246799d748110a8f4e188" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91d0b45aa09e4a418bc838d6ad0414b0"
I20260619 18:51:55.166702 24186 raft_consensus.cc:3037] T 6e1495389b1d4657826ca244d93f6626 P 49d19883f290401f82b63c29f0c1b3a8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:51:55.166766 24055 raft_consensus.cc:3037] T 6e1495389b1d4657826ca244d93f6626 P 91d0b45aa09e4a418bc838d6ad0414b0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:51:55.169925 24055 raft_consensus.cc:2454] T 6e1495389b1d4657826ca244d93f6626 P 91d0b45aa09e4a418bc838d6ad0414b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2e61ba10bc246799d748110a8f4e188 in term 1.
I20260619 18:51:55.169925 24186 raft_consensus.cc:2454] T 6e1495389b1d4657826ca244d93f6626 P 49d19883f290401f82b63c29f0c1b3a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2e61ba10bc246799d748110a8f4e188 in term 1.
I20260619 18:51:55.170336 24252 leader_election.cc:302] T 6e1495389b1d4657826ca244d93f6626 P d2e61ba10bc246799d748110a8f4e188 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91d0b45aa09e4a418bc838d6ad0414b0, d2e61ba10bc246799d748110a8f4e188; no voters:
I20260619 18:51:55.170634 24385 raft_consensus.cc:2781] T 6e1495389b1d4657826ca244d93f6626 P d2e61ba10bc246799d748110a8f4e188 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:51:55.171051 24385 raft_consensus.cc:686] T 6e1495389b1d4657826ca244d93f6626 P d2e61ba10bc246799d748110a8f4e188 [term 1 LEADER]: Becoming Leader. State: Replica: d2e61ba10bc246799d748110a8f4e188, State: Running, Role: LEADER
I20260619 18:51:55.171584 24385 consensus_queue.cc:237] T 6e1495389b1d4657826ca244d93f6626 P d2e61ba10bc246799d748110a8f4e188 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49d19883f290401f82b63c29f0c1b3a8" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 46335 } } peers { permanent_uuid: "91d0b45aa09e4a418bc838d6ad0414b0" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 34057 } } peers { permanent_uuid: "d2e61ba10bc246799d748110a8f4e188" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40897 } }
I20260619 18:51:55.175909 23914 catalog_manager.cc:5495] T 6e1495389b1d4657826ca244d93f6626 P d2e61ba10bc246799d748110a8f4e188 reported cstate change: term changed from 0 to 1, leader changed from <none> to d2e61ba10bc246799d748110a8f4e188 (127.3.243.67). New cstate: current_term: 1 leader_uuid: "d2e61ba10bc246799d748110a8f4e188" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49d19883f290401f82b63c29f0c1b3a8" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 46335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91d0b45aa09e4a418bc838d6ad0414b0" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 34057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d2e61ba10bc246799d748110a8f4e188" member_type: VOTER last_known_addr { host: "127.3.243.67" port: 40897 } health_report { overall_health: HEALTHY } } }
I20260619 18:51:55.192088 24377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:55.186894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38410 (local address 127.3.243.67:40897)
0619 18:51:55.187078 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 18:51:55.187084 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:51:55.187103 (+ 19us) server_negotiation.cc:407] Connection header received
0619 18:51:55.187310 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:51:55.187314 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:51:55.187395 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:51:55.187518 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:51:55.187863 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:55.188627 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:55.189992 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:55.190426 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:55.190533 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:55.190997 (+ 464us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:51:55.191528 (+ 531us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:51:55.191660 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0619 18:51:55.191743 (+ 83us) server_negotiation.cc:299] Negotiation successful
0619 18:51:55.191861 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 18:51:55.200044 24055 raft_consensus.cc:1261] T 6e1495389b1d4657826ca244d93f6626 P 91d0b45aa09e4a418bc838d6ad0414b0 [term 1 FOLLOWER]: Refusing update from remote peer d2e61ba10bc246799d748110a8f4e188: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:51:55.200044 24186 raft_consensus.cc:1261] T 6e1495389b1d4657826ca244d93f6626 P 49d19883f290401f82b63c29f0c1b3a8 [term 1 FOLLOWER]: Refusing update from remote peer d2e61ba10bc246799d748110a8f4e188: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:51:55.201133 24385 consensus_queue.cc:1035] T 6e1495389b1d4657826ca244d93f6626 P d2e61ba10bc246799d748110a8f4e188 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d0b45aa09e4a418bc838d6ad0414b0" member_type: VOTER last_known_addr { host: "127.3.243.65" port: 34057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:51:55.201437 24389 consensus_queue.cc:1035] T 6e1495389b1d4657826ca244d93f6626 P d2e61ba10bc246799d748110a8f4e188 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49d19883f290401f82b63c29f0c1b3a8" member_type: VOTER last_known_addr { host: "127.3.243.66" port: 46335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:51:55.226432 23914 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49800:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:51:55.226698 23914 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49800:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:51:55.230871 23914 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 532a05d0408e41b0b9eb76b923f68d2a: Sending DeleteTablet for 3 replicas of tablet 6e1495389b1d4657826ca244d93f6626
I20260619 18:51:55.231994 24166 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e1495389b1d4657826ca244d93f6626 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:51:55 UTC) from {username='slave'} at 127.0.0.1:37570
I20260619 18:51:55.232066 24035 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e1495389b1d4657826ca244d93f6626 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:51:55 UTC) from {username='slave'} at 127.0.0.1:36800
I20260619 18:51:55.232887 24296 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e1495389b1d4657826ca244d93f6626 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:51:55 UTC) from {username='slave'} at 127.0.0.1:38408
I20260619 18:51:55.233700 24404 tablet_replica.cc:330] T 6e1495389b1d4657826ca244d93f6626 P d2e61ba10bc246799d748110a8f4e188: stopping tablet replica
I20260619 18:51:55.233752 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 23973
I20260619 18:51:55.234272 24404 raft_consensus.cc:2229] T 6e1495389b1d4657826ca244d93f6626 P d2e61ba10bc246799d748110a8f4e188 [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:51:55.235071 24404 raft_consensus.cc:2258] T 6e1495389b1d4657826ca244d93f6626 P d2e61ba10bc246799d748110a8f4e188 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:51:55.237411 24404 ts_tablet_manager.cc:1905] T 6e1495389b1d4657826ca244d93f6626 P d2e61ba10bc246799d748110a8f4e188: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:51:55.243261 24403 tablet_replica.cc:330] T 6e1495389b1d4657826ca244d93f6626 P 91d0b45aa09e4a418bc838d6ad0414b0: stopping tablet replica
I20260619 18:51:55.243701 24403 raft_consensus.cc:2229] T 6e1495389b1d4657826ca244d93f6626 P 91d0b45aa09e4a418bc838d6ad0414b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:51:55.244266 24403 raft_consensus.cc:2258] T 6e1495389b1d4657826ca244d93f6626 P 91d0b45aa09e4a418bc838d6ad0414b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:51:55.245079 24402 tablet_replica.cc:330] T 6e1495389b1d4657826ca244d93f6626 P 49d19883f290401f82b63c29f0c1b3a8: stopping tablet replica
I20260619 18:51:55.245471 24402 raft_consensus.cc:2229] T 6e1495389b1d4657826ca244d93f6626 P 49d19883f290401f82b63c29f0c1b3a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:51:55.245944 24402 raft_consensus.cc:2258] T 6e1495389b1d4657826ca244d93f6626 P 49d19883f290401f82b63c29f0c1b3a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:51:55.246565 24404 ts_tablet_manager.cc:1918] T 6e1495389b1d4657826ca244d93f6626 P d2e61ba10bc246799d748110a8f4e188: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:51:55.246779 24404 log.cc:1196] T 6e1495389b1d4657826ca244d93f6626 P d2e61ba10bc246799d748110a8f4e188: Deleting WAL directory at /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894810715282-4045-0/minicluster-data/ts-2/wal/wals/6e1495389b1d4657826ca244d93f6626
I20260619 18:51:55.247432 24404 ts_tablet_manager.cc:1939] T 6e1495389b1d4657826ca244d93f6626 P d2e61ba10bc246799d748110a8f4e188: Deleting consensus metadata
I20260619 18:51:55.248016 24403 ts_tablet_manager.cc:1905] T 6e1495389b1d4657826ca244d93f6626 P 91d0b45aa09e4a418bc838d6ad0414b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:51:55.249609 23900 catalog_manager.cc:4841] TS d2e61ba10bc246799d748110a8f4e188 (127.3.243.67:40897): tablet 6e1495389b1d4657826ca244d93f6626 (table test-table [id=9baf262c37d04b4abf95b0fdc4c4adb6]) successfully deleted
I20260619 18:51:55.253767 24402 ts_tablet_manager.cc:1905] T 6e1495389b1d4657826ca244d93f6626 P 49d19883f290401f82b63c29f0c1b3a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:51:55.254923 24403 ts_tablet_manager.cc:1918] T 6e1495389b1d4657826ca244d93f6626 P 91d0b45aa09e4a418bc838d6ad0414b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:51:55.255076 24403 log.cc:1196] T 6e1495389b1d4657826ca244d93f6626 P 91d0b45aa09e4a418bc838d6ad0414b0: Deleting WAL directory at /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894810715282-4045-0/minicluster-data/ts-0/wal/wals/6e1495389b1d4657826ca244d93f6626
I20260619 18:51:55.255580 24403 ts_tablet_manager.cc:1939] T 6e1495389b1d4657826ca244d93f6626 P 91d0b45aa09e4a418bc838d6ad0414b0: Deleting consensus metadata
I20260619 18:51:55.257434 23901 catalog_manager.cc:4841] TS 91d0b45aa09e4a418bc838d6ad0414b0 (127.3.243.65:34057): tablet 6e1495389b1d4657826ca244d93f6626 (table test-table [id=9baf262c37d04b4abf95b0fdc4c4adb6]) successfully deleted
I20260619 18:51:55.259949 24378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:55.236711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36810 (local address 127.3.243.65:34057)
0619 18:51:55.236909 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 18:51:55.236915 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:51:55.251487 (+ 14572us) server_negotiation.cc:407] Connection header received
0619 18:51:55.251673 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:51:55.251679 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:51:55.251794 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:51:55.251893 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:51:55.252882 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:55.254002 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:55.255798 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:55.256480 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:55.256574 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:55.258290 (+ 1716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:51:55.258300 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:51:55.258317 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:51:55.259083 (+ 766us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:51:55.259144 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:51:55.259169 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:51:55.259279 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 18:51:55.259484 (+ 205us) server_negotiation.cc:299] Negotiation successful
0619 18:51:55.259599 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260619 18:51:55.260726 24402 ts_tablet_manager.cc:1918] T 6e1495389b1d4657826ca244d93f6626 P 49d19883f290401f82b63c29f0c1b3a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:51:55.260902 24402 log.cc:1196] T 6e1495389b1d4657826ca244d93f6626 P 49d19883f290401f82b63c29f0c1b3a8: Deleting WAL directory at /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894810715282-4045-0/minicluster-data/ts-1/wal/wals/6e1495389b1d4657826ca244d93f6626
I20260619 18:51:55.261224 24095 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:51:55.261319 24402 ts_tablet_manager.cc:1939] T 6e1495389b1d4657826ca244d93f6626 P 49d19883f290401f82b63c29f0c1b3a8: Deleting consensus metadata
I20260619 18:51:55.263258 23899 catalog_manager.cc:4841] TS 49d19883f290401f82b63c29f0c1b3a8 (127.3.243.66:46335): tablet 6e1495389b1d4657826ca244d93f6626 (table test-table [id=9baf262c37d04b4abf95b0fdc4c4adb6]) successfully deleted
I20260619 18:51:55.349677 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 23973
I20260619 18:51:55.365116 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 24104
I20260619 18:51:55.372107 24375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:55.366028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37578 (local address 127.3.243.66:46335)
0619 18:51:55.366203 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 18:51:55.366210 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:51:55.366338 (+ 128us) server_negotiation.cc:407] Connection header received
0619 18:51:55.366528 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:51:55.366533 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:51:55.366618 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:51:55.366705 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:51:55.367115 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:55.367870 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:55.369167 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:55.369568 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:55.369681 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:55.370756 (+ 1075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:51:55.370765 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:51:55.370787 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:51:55.371375 (+ 588us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:51:55.371428 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:51:55.371448 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:51:55.371546 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 18:51:55.371683 (+ 137us) server_negotiation.cc:299] Negotiation successful
0619 18:51:55.371811 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 18:51:55.372915 24226 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:51:55.460179 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 24104
I20260619 18:51:55.474167 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 24234
I20260619 18:51:55.482002 24377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:55.474929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38422 (local address 127.3.243.67:40897)
0619 18:51:55.475115 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 18:51:55.475119 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:51:55.475164 (+ 45us) server_negotiation.cc:407] Connection header received
0619 18:51:55.475376 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:51:55.475380 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:51:55.475455 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:51:55.475534 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:51:55.476023 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:55.476890 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:55.478691 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:55.479129 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:55.479216 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:55.480491 (+ 1275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:51:55.480502 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:51:55.480518 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:51:55.481221 (+ 703us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:51:55.481272 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:51:55.481305 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:51:55.481383 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 18:51:55.481561 (+ 178us) server_negotiation.cc:299] Negotiation successful
0619 18:51:55.481649 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 18:51:55.482790 24356 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:51:55.603879 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 24234
I20260619 18:51:55.619910 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 23883
I20260619 18:51:55.621059 23944 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:51:55.728673 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 23883
2026-06-19T18:51:55Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2677 ms)
[----------] 6 tests from EncryptionPolicyTest (19893 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781894810715282-4045-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:51:55 dist-test-slave-m9wg krb5kdc[24412](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:51:55 dist-test-slave-m9wg krb5kdc[24412](info): set up 2 sockets
Jun 19 18:51:55 dist-test-slave-m9wg krb5kdc[24412](info): commencing operation
krb5kdc: starting...
W20260619 18:51:56.732424 4045 mini_kdc.cc:121] Time spent starting KDC: real 0.963s user 0.003s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781894810715282-4045-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781894810715282-4045-0/krb5kdc/test-user.keytab.
Jun 19 18:51:56 dist-test-slave-m9wg krb5kdc[24412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895116, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:51:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:51:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781894810715282-4045-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781894810715282-4045-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 18:51:57.020004 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781894810715282-4045-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781894810715282-4045-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781894810715282-4045-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37361
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781894810715282-4045-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37361
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:51:57.399432 24425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:51:57.399809 24425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:51:57.399937 24425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:51:57.408010 24425 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:51:57.408231 24425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:51:57.408329 24425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:51:57.408419 24425 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:51:57.408479 24425 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:51:57.419652 24425 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781894810715282-4045-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781894810715282-4045-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37361
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781894810715282-4045-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37361
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781894810715282-4045-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:51:57.421332 24425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:51:57.422904 24425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:51:57.436298 24430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:51:57.436329 24431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:51:57.436228 24433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:51:57.437822 24425 server_base.cc:1034] running on GCE node
I20260619 18:51:57.438866 24425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:51:57.440781 24425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:51:57.442003 24425 hybrid_clock.cc:648] HybridClock initialized: now 1781895117441945 us; error 60 us; skew 500 ppm
I20260619 18:51:57.442417 24425 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:51:57 dist-test-slave-m9wg krb5kdc[24412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895117, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:51:57.445943 24425 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 18:51:57.447731 24425 webserver.cc:460] Webserver started at http://127.0.0.1:33379/ using document root <none> and password file <none>
I20260619 18:51:57.448650 24425 fs_manager.cc:359] Metadata directory not provided
I20260619 18:51:57.448788 24425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:51:57.449133 24425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:51:57.452890 24425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781894810715282-4045-0/minicluster-data/master-0/data/instance:
uuid: "561a354b0a5148688eacc3fcfb7d58d9"
format_stamp: "Formatted at 2026-06-19 18:51:57 on dist-test-slave-m9wg"
server_key: "3eeef32e2087941e0fd5039e0733a680"
server_key_iv: "97052d0a35442c0c2a3422a06343bf83"
server_key_version: "encryptionkey@0"
I20260619 18:51:57.453892 24425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781894810715282-4045-0/minicluster-data/master-0/wal/instance:
uuid: "561a354b0a5148688eacc3fcfb7d58d9"
format_stamp: "Formatted at 2026-06-19 18:51:57 on dist-test-slave-m9wg"
server_key: "3eeef32e2087941e0fd5039e0733a680"
server_key_iv: "97052d0a35442c0c2a3422a06343bf83"
server_key_version: "encryptionkey@0"
I20260619 18:51:57.460935 24425 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260619 18:51:57.465031 24440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:51:57.466585 24425 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 18:51:57.466761 24425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781894810715282-4045-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781894810715282-4045-0/minicluster-data/master-0/wal
uuid: "561a354b0a5148688eacc3fcfb7d58d9"
format_stamp: "Formatted at 2026-06-19 18:51:57 on dist-test-slave-m9wg"
server_key: "3eeef32e2087941e0fd5039e0733a680"
server_key_iv: "97052d0a35442c0c2a3422a06343bf83"
server_key_version: "encryptionkey@0"
I20260619 18:51:57.466966 24425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781894810715282-4045-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781894810715282-4045-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781894810715282-4045-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:51:57.495265 24425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:51:57.499620 24425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:51:57.499907 24425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:51:57.519035 24425 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37361
I20260619 18:51:57.519061 24491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37361 every 8 connection(s)
I20260619 18:51:57.520620 24425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
I20260619 18:51:57.521049 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 24425
I20260619 18:51:57.521257 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781894810715282-4045-0/minicluster-data/master-0/wal/instance
I20260619 18:51:57.522120 4045 external_mini_cluster.cc:1428] Setting key 14c4d9040aadbe3425ff29b42d198caa
I20260619 18:51:57.525883 24492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 18:51:57 dist-test-slave-m9wg krb5kdc[24412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895116, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 18:51:57.539937 24492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 561a354b0a5148688eacc3fcfb7d58d9: Bootstrap starting.
I20260619 18:51:57.544661 24492 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 561a354b0a5148688eacc3fcfb7d58d9: Neither blocks nor log segments found. Creating new log.
I20260619 18:51:57.546108 24492 log.cc:824] T 00000000000000000000000000000000 P 561a354b0a5148688eacc3fcfb7d58d9: Log is configured to *not* fsync() on all Append() calls
I20260619 18:51:57.549628 24495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:57.524445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47647 (local address 127.0.0.1:37361)
0619 18:51:57.525265 (+ 820us) server_negotiation.cc:206] Beginning negotiation
0619 18:51:57.525283 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 18:51:57.525340 (+ 57us) server_negotiation.cc:407] Connection header received
0619 18:51:57.527749 (+ 2409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:51:57.527779 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:51:57.528370 (+ 591us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:51:57.528842 (+ 472us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:51:57.530338 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:57.532620 (+ 2282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:57.534103 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:57.535132 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:57.535316 (+ 184us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:57.539869 (+ 4553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:51:57.539880 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:51:57.539901 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:51:57.539963 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:51:57.543949 (+ 3986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:51:57.545097 (+ 1148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:51:57.545108 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:51:57.545122 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:51:57.545348 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:51:57.545912 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:51:57.545917 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:51:57.545920 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:51:57.546737 (+ 817us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:51:57.547078 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0619 18:51:57.547710 (+ 632us) server_negotiation.cc:299] Negotiation successful
0619 18:51:57.548108 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":337,"threads_started":1}
I20260619 18:51:57.550225 24492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 561a354b0a5148688eacc3fcfb7d58d9: No bootstrap required, opened a new log
I20260619 18:51:57.554383 4045 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:51:57.559053 24492 raft_consensus.cc:348] T 00000000000000000000000000000000 P 561a354b0a5148688eacc3fcfb7d58d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "561a354b0a5148688eacc3fcfb7d58d9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37361 } }
I20260619 18:51:57.559608 24492 raft_consensus.cc:374] T 00000000000000000000000000000000 P 561a354b0a5148688eacc3fcfb7d58d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:51:57.559720 24492 raft_consensus.cc:729] T 00000000000000000000000000000000 P 561a354b0a5148688eacc3fcfb7d58d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 561a354b0a5148688eacc3fcfb7d58d9, State: Initialized, Role: FOLLOWER
I20260619 18:51:57.560664 24492 consensus_queue.cc:260] T 00000000000000000000000000000000 P 561a354b0a5148688eacc3fcfb7d58d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "561a354b0a5148688eacc3fcfb7d58d9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37361 } }
I20260619 18:51:57.560911 24492 raft_consensus.cc:388] T 00000000000000000000000000000000 P 561a354b0a5148688eacc3fcfb7d58d9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:51:57.561036 24492 raft_consensus.cc:482] T 00000000000000000000000000000000 P 561a354b0a5148688eacc3fcfb7d58d9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:51:57.561206 24492 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 561a354b0a5148688eacc3fcfb7d58d9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:51:57.563616 24492 raft_consensus.cc:504] T 00000000000000000000000000000000 P 561a354b0a5148688eacc3fcfb7d58d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "561a354b0a5148688eacc3fcfb7d58d9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37361 } }
I20260619 18:51:57.564178 24492 leader_election.cc:302] T 00000000000000000000000000000000 P 561a354b0a5148688eacc3fcfb7d58d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 561a354b0a5148688eacc3fcfb7d58d9; no voters:
I20260619 18:51:57.564885 24492 leader_election.cc:288] T 00000000000000000000000000000000 P 561a354b0a5148688eacc3fcfb7d58d9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:51:57.564985 24502 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 561a354b0a5148688eacc3fcfb7d58d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:51:57.566363 24502 raft_consensus.cc:686] T 00000000000000000000000000000000 P 561a354b0a5148688eacc3fcfb7d58d9 [term 1 LEADER]: Becoming Leader. State: Replica: 561a354b0a5148688eacc3fcfb7d58d9, State: Running, Role: LEADER
I20260619 18:51:57.566905 24502 consensus_queue.cc:237] T 00000000000000000000000000000000 P 561a354b0a5148688eacc3fcfb7d58d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "561a354b0a5148688eacc3fcfb7d58d9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37361 } }
I20260619 18:51:57.567574 24492 sys_catalog.cc:564] T 00000000000000000000000000000000 P 561a354b0a5148688eacc3fcfb7d58d9 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:51:57.570024 24503 sys_catalog.cc:455] T 00000000000000000000000000000000 P 561a354b0a5148688eacc3fcfb7d58d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "561a354b0a5148688eacc3fcfb7d58d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "561a354b0a5148688eacc3fcfb7d58d9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37361 } } }
I20260619 18:51:57.570397 24503 sys_catalog.cc:458] T 00000000000000000000000000000000 P 561a354b0a5148688eacc3fcfb7d58d9 [sys.catalog]: This master's current role is: LEADER
I20260619 18:51:57.569636 24504 sys_catalog.cc:455] T 00000000000000000000000000000000 P 561a354b0a5148688eacc3fcfb7d58d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 561a354b0a5148688eacc3fcfb7d58d9. Latest consensus state: current_term: 1 leader_uuid: "561a354b0a5148688eacc3fcfb7d58d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "561a354b0a5148688eacc3fcfb7d58d9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37361 } } }
I20260619 18:51:57.572412 24504 sys_catalog.cc:458] T 00000000000000000000000000000000 P 561a354b0a5148688eacc3fcfb7d58d9 [sys.catalog]: This master's current role is: LEADER
I20260619 18:51:57.573817 24507 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:51:57.579172 24495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:57.559477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51903 (local address 127.0.0.1:37361)
0619 18:51:57.559663 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 18:51:57.559668 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:51:57.560367 (+ 699us) server_negotiation.cc:407] Connection header received
0619 18:51:57.560553 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:51:57.560559 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:51:57.560624 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:51:57.560711 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:51:57.562261 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:57.563095 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:57.564436 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:57.565210 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:57.566017 (+ 807us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:57.568620 (+ 2603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:51:57.568627 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:51:57.568632 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:51:57.568673 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:51:57.574434 (+ 5761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:51:57.575471 (+ 1037us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:51:57.575476 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:51:57.575479 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:51:57.575694 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:51:57.576372 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:51:57.576377 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:51:57.576379 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:51:57.576835 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:51:57.578260 (+ 1425us) server_negotiation.cc:1035] Waiting for connection context
0619 18:51:57.578462 (+ 202us) server_negotiation.cc:299] Negotiation successful
0619 18:51:57.578676 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260619 18:51:57.589476 24507 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:51:57.600021 24507 catalog_manager.cc:1347] Generated new cluster ID: e204f8110e934debbfbd0f8e4d52e590
I20260619 18:51:57.600180 24507 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:51:57.611172 24507 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:51:57.612694 24507 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:51:57.620699 24507 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 561a354b0a5148688eacc3fcfb7d58d9: Generated new TSK 0
I20260619 18:51:57.621744 24507 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:51:57.632835 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 24425
I20260619 18:51:57.634665 24487 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:51:57.733204 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 24425
W20260619 18:51:57.751300 24410 connection.cc:535] client connection to 127.0.0.1:37361 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37361): Connection reset by peer (error 104)
2026-06-19T18:51:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2001 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781894810715282-4045-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:51:57 dist-test-slave-m9wg krb5kdc[24524](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:51:57 dist-test-slave-m9wg krb5kdc[24524](info): set up 2 sockets
Jun 19 18:51:57 dist-test-slave-m9wg krb5kdc[24524](info): commencing operation
krb5kdc: starting...
W20260619 18:51:58.768594 4045 mini_kdc.cc:121] Time spent starting KDC: real 0.997s user 0.000s sys 0.036s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781894810715282-4045-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781894810715282-4045-0/krb5kdc/test-user.keytab.
Jun 19 18:51:58 dist-test-slave-m9wg krb5kdc[24524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895118, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:51:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:51:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781894810715282-4045-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781894810715282-4045-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 18:51:59.051576 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781894810715282-4045-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781894810715282-4045-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781894810715282-4045-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35913
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781894810715282-4045-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35913
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:51:59.442909 24537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:51:59.443250 24537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:51:59.443393 24537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:51:59.450865 24537 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:51:59.451002 24537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:51:59.451084 24537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:51:59.451169 24537 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:51:59.451228 24537 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:51:59.461740 24537 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781894810715282-4045-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781894810715282-4045-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35913
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781894810715282-4045-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35913
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781894810715282-4045-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:51:59.463296 24537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:51:59.464875 24537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:51:59.477957 24542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:51:59.477880 24545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:51:59.477857 24543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:51:59.478513 24537 server_base.cc:1034] running on GCE node
I20260619 18:51:59.479453 24537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:51:59.481179 24537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:51:59.482425 24537 hybrid_clock.cc:648] HybridClock initialized: now 1781895119482381 us; error 40 us; skew 500 ppm
I20260619 18:51:59.482890 24537 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:51:59 dist-test-slave-m9wg krb5kdc[24524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895119, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:51:59.486745 24537 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 18:51:59.488664 24537 webserver.cc:460] Webserver started at http://127.0.0.1:46457/ using document root <none> and password file <none>
I20260619 18:51:59.489532 24537 fs_manager.cc:359] Metadata directory not provided
I20260619 18:51:59.489660 24537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:51:59.489987 24537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:51:59.493619 24537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781894810715282-4045-0/minicluster-data/master-0/data/instance:
uuid: "8ad2370d1aab45cc83145659d04883a7"
format_stamp: "Formatted at 2026-06-19 18:51:59 on dist-test-slave-m9wg"
server_key: "058480fd9cf8c718ffc8e00b2c92362d"
server_key_iv: "4def923530719c511296e96191a0b9d6"
server_key_version: "encryptionkey@0"
I20260619 18:51:59.494510 24537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781894810715282-4045-0/minicluster-data/master-0/wal/instance:
uuid: "8ad2370d1aab45cc83145659d04883a7"
format_stamp: "Formatted at 2026-06-19 18:51:59 on dist-test-slave-m9wg"
server_key: "058480fd9cf8c718ffc8e00b2c92362d"
server_key_iv: "4def923530719c511296e96191a0b9d6"
server_key_version: "encryptionkey@0"
I20260619 18:51:59.500891 24537 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260619 18:51:59.505329 24552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:51:59.507192 24537 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.003s
I20260619 18:51:59.507458 24537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781894810715282-4045-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781894810715282-4045-0/minicluster-data/master-0/wal
uuid: "8ad2370d1aab45cc83145659d04883a7"
format_stamp: "Formatted at 2026-06-19 18:51:59 on dist-test-slave-m9wg"
server_key: "058480fd9cf8c718ffc8e00b2c92362d"
server_key_iv: "4def923530719c511296e96191a0b9d6"
server_key_version: "encryptionkey@0"
I20260619 18:51:59.507716 24537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781894810715282-4045-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781894810715282-4045-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781894810715282-4045-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:51:59.527987 24537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:51:59.532367 24537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:51:59.532648 24537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:51:59.551749 24537 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35913
I20260619 18:51:59.551770 24603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35913 every 8 connection(s)
I20260619 18:51:59.553448 24537 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
I20260619 18:51:59.557862 24604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:51:59.562702 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 24537
I20260619 18:51:59.562906 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781894810715282-4045-0/minicluster-data/master-0/wal/instance
I20260619 18:51:59.563494 4045 external_mini_cluster.cc:1428] Setting key 2faeaad7b6d2ed32d5e2ca2106b81c07
I20260619 18:51:59.570353 24604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ad2370d1aab45cc83145659d04883a7: Bootstrap starting.
I20260619 18:51:59.573840 24604 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ad2370d1aab45cc83145659d04883a7: Neither blocks nor log segments found. Creating new log.
I20260619 18:51:59.575165 24604 log.cc:824] T 00000000000000000000000000000000 P 8ad2370d1aab45cc83145659d04883a7: Log is configured to *not* fsync() on all Append() calls
I20260619 18:51:59.578243 24607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:59.565688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58199 (local address 127.0.0.1:35913)
0619 18:51:59.566533 (+ 845us) server_negotiation.cc:206] Beginning negotiation
0619 18:51:59.566547 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 18:51:59.566595 (+ 48us) server_negotiation.cc:407] Connection header received
0619 18:51:59.568963 (+ 2368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:51:59.568991 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:51:59.569456 (+ 465us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:51:59.569848 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:51:59.570484 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:59.571878 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:59.573300 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:59.574072 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:59.574291 (+ 219us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:59.575565 (+ 1274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:51:59.575573 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:51:59.575592 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:51:59.576466 (+ 874us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:51:59.576571 (+ 105us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:51:59.576605 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:51:59.576745 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 18:51:59.577036 (+ 291us) server_negotiation.cc:299] Negotiation successful
0619 18:51:59.577211 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":372,"threads_started":1}
I20260619 18:51:59.578336 24604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ad2370d1aab45cc83145659d04883a7: No bootstrap required, opened a new log
I20260619 18:51:59.582546 4045 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:51:59.587011 24604 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ad2370d1aab45cc83145659d04883a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ad2370d1aab45cc83145659d04883a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35913 } }
I20260619 18:51:59.587564 24604 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8ad2370d1aab45cc83145659d04883a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:51:59.587687 24604 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ad2370d1aab45cc83145659d04883a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ad2370d1aab45cc83145659d04883a7, State: Initialized, Role: FOLLOWER
I20260619 18:51:59.588547 24604 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ad2370d1aab45cc83145659d04883a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ad2370d1aab45cc83145659d04883a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35913 } }
I20260619 18:51:59.588878 24604 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8ad2370d1aab45cc83145659d04883a7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:51:59.589032 24604 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ad2370d1aab45cc83145659d04883a7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:51:59.589221 24604 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ad2370d1aab45cc83145659d04883a7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:51:59.592417 24604 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ad2370d1aab45cc83145659d04883a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ad2370d1aab45cc83145659d04883a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35913 } }
I20260619 18:51:59.593196 24604 leader_election.cc:302] T 00000000000000000000000000000000 P 8ad2370d1aab45cc83145659d04883a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8ad2370d1aab45cc83145659d04883a7; no voters:
I20260619 18:51:59.594120 24604 leader_election.cc:288] T 00000000000000000000000000000000 P 8ad2370d1aab45cc83145659d04883a7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:51:59.594236 24614 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ad2370d1aab45cc83145659d04883a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:51:59.595047 24607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:51:59.587242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53607 (local address 127.0.0.1:35913)
0619 18:51:59.587464 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0619 18:51:59.587470 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:51:59.588080 (+ 610us) server_negotiation.cc:407] Connection header received
0619 18:51:59.588336 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:51:59.588342 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:51:59.588433 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:51:59.588575 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:51:59.589260 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:59.590428 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:59.591860 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:51:59.592593 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:51:59.592722 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:51:59.594088 (+ 1366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:51:59.594096 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:51:59.594100 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:51:59.594126 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:51:59.594175 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:51:59.594194 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:51:59.594312 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 18:51:59.594599 (+ 287us) server_negotiation.cc:299] Negotiation successful
0619 18:51:59.594667 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 18:51:59.596467 24614 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8ad2370d1aab45cc83145659d04883a7 [term 1 LEADER]: Becoming Leader. State: Replica: 8ad2370d1aab45cc83145659d04883a7, State: Running, Role: LEADER
I20260619 18:51:59.597163 24614 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ad2370d1aab45cc83145659d04883a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ad2370d1aab45cc83145659d04883a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35913 } }
I20260619 18:51:59.597433 24604 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ad2370d1aab45cc83145659d04883a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:51:59.600483 24615 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ad2370d1aab45cc83145659d04883a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8ad2370d1aab45cc83145659d04883a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ad2370d1aab45cc83145659d04883a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35913 } } }
I20260619 18:51:59.600594 24616 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ad2370d1aab45cc83145659d04883a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ad2370d1aab45cc83145659d04883a7. Latest consensus state: current_term: 1 leader_uuid: "8ad2370d1aab45cc83145659d04883a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ad2370d1aab45cc83145659d04883a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35913 } } }
I20260619 18:51:59.600980 24615 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ad2370d1aab45cc83145659d04883a7 [sys.catalog]: This master's current role is: LEADER
I20260619 18:51:59.601015 24616 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ad2370d1aab45cc83145659d04883a7 [sys.catalog]: This master's current role is: LEADER
I20260619 18:51:59.608867 24623 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:51:59.615008 24623 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:51:59.626323 24623 catalog_manager.cc:1347] Generated new cluster ID: 03c10b39e2cd45e5b48bb14409fcdcbc
I20260619 18:51:59.626472 24623 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:51:59.639824 24623 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:51:59.641409 24623 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:51:59.653317 24623 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8ad2370d1aab45cc83145659d04883a7: Generated new TSK 0
I20260619 18:51:59.654506 24623 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:51:59.664040 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 24537
W20260619 18:51:59.665959 24599 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:58199
W20260619 18:51:59.666931 4045 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 24537: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 18:51:59.667148 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 24537
W20260619 18:51:59.684897 24522 connection.cc:535] client connection to 127.0.0.1:35913 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35913): Connection reset by peer (error 104)
2026-06-19T18:51:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1933 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781894810715282-4045-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:51:59 dist-test-slave-m9wg krb5kdc[24635](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:51:59 dist-test-slave-m9wg krb5kdc[24635](info): set up 2 sockets
Jun 19 18:51:59 dist-test-slave-m9wg krb5kdc[24635](info): commencing operation
krb5kdc: starting...
W20260619 18:52:00.716902 4045 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.003s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781894810715282-4045-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781894810715282-4045-0/krb5kdc/test-user.keytab.
Jun 19 18:52:00 dist-test-slave-m9wg krb5kdc[24635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895120, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:52:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:52:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781894810715282-4045-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781894810715282-4045-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 18:52:01.002130 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781894810715282-4045-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781894810715282-4045-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781894810715282-4045-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34449
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781894810715282-4045-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34449
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:52:01.388978 24648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:52:01.389454 24648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:52:01.389611 24648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:52:01.398365 24648 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:52:01.398571 24648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:52:01.398653 24648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:52:01.398761 24648 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:52:01.398840 24648 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:52:01.411311 24648 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781894810715282-4045-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781894810715282-4045-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34449
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781894810715282-4045-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781894810715282-4045-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:52:01.413216 24648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:52:01.415046 24648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:52:01.428856 24653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:52:01.428828 24656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:52:01.428797 24654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:52:01.430092 24648 server_base.cc:1034] running on GCE node
I20260619 18:52:01.430980 24648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:52:01.432701 24648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:52:01.433920 24648 hybrid_clock.cc:648] HybridClock initialized: now 1781895121433866 us; error 42 us; skew 500 ppm
I20260619 18:52:01.434301 24648 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:52:01 dist-test-slave-m9wg krb5kdc[24635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895121, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:52:01.437796 24648 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 18:52:01.439321 24648 webserver.cc:460] Webserver started at http://127.0.0.1:35445/ using document root <none> and password file <none>
I20260619 18:52:01.440063 24648 fs_manager.cc:359] Metadata directory not provided
I20260619 18:52:01.440189 24648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:52:01.440505 24648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:52:01.443781 24648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781894810715282-4045-0/minicluster-data/master-0/data/instance:
uuid: "c868847a3c9f4d65a1d21045159c1bf0"
format_stamp: "Formatted at 2026-06-19 18:52:01 on dist-test-slave-m9wg"
server_key: "8de37e7225237bf6923baee8e753816b"
server_key_iv: "ad70cdf31859f83569b9d9e4869b0d03"
server_key_version: "encryptionkey@0"
I20260619 18:52:01.444586 24648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781894810715282-4045-0/minicluster-data/master-0/wal/instance:
uuid: "c868847a3c9f4d65a1d21045159c1bf0"
format_stamp: "Formatted at 2026-06-19 18:52:01 on dist-test-slave-m9wg"
server_key: "8de37e7225237bf6923baee8e753816b"
server_key_iv: "ad70cdf31859f83569b9d9e4869b0d03"
server_key_version: "encryptionkey@0"
I20260619 18:52:01.450165 24648 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 18:52:01.454150 24663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:52:01.455827 24648 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 18:52:01.456007 24648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781894810715282-4045-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781894810715282-4045-0/minicluster-data/master-0/wal
uuid: "c868847a3c9f4d65a1d21045159c1bf0"
format_stamp: "Formatted at 2026-06-19 18:52:01 on dist-test-slave-m9wg"
server_key: "8de37e7225237bf6923baee8e753816b"
server_key_iv: "ad70cdf31859f83569b9d9e4869b0d03"
server_key_version: "encryptionkey@0"
I20260619 18:52:01.456255 24648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781894810715282-4045-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781894810715282-4045-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781894810715282-4045-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:52:01.462698 24648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:52:01.467247 24648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:52:01.467547 24648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:52:01.487839 24648 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34449
I20260619 18:52:01.487886 24714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34449 every 8 connection(s)
I20260619 18:52:01.489581 24648 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
I20260619 18:52:01.493552 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 24648
I20260619 18:52:01.493757 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781894810715282-4045-0/minicluster-data/master-0/wal/instance
I20260619 18:52:01.494297 4045 external_mini_cluster.cc:1428] Setting key a7c954580f0951dcb81184c2cd79ab41
I20260619 18:52:01.494426 24715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:52:01.503571 24718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:01.496332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33915 (local address 127.0.0.1:34449)
0619 18:52:01.497179 (+ 847us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:01.497193 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:01.497240 (+ 47us) server_negotiation.cc:407] Connection header received
0619 18:52:01.499689 (+ 2449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:01.499717 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:01.500142 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:01.500614 (+ 472us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:01.501823 (+ 1209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:52:01.501829 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:52:01.501847 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:52:01.501861 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:52:01.501933 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:52:01.501954 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:52:01.502143 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:01.502254 (+ 111us) server_negotiation.cc:299] Negotiation successful
0619 18:52:01.502524 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":370,"threads_started":1}
I20260619 18:52:01.507750 4045 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:52:01.509722 24715 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c868847a3c9f4d65a1d21045159c1bf0: Bootstrap starting.
I20260619 18:52:01.513911 24715 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c868847a3c9f4d65a1d21045159c1bf0: Neither blocks nor log segments found. Creating new log.
I20260619 18:52:01.515210 24715 log.cc:824] T 00000000000000000000000000000000 P c868847a3c9f4d65a1d21045159c1bf0: Log is configured to *not* fsync() on all Append() calls
I20260619 18:52:01.515620 24718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:01.513251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36089 (local address 127.0.0.1:34449)
0619 18:52:01.513424 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:01.513430 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:01.513827 (+ 397us) server_negotiation.cc:407] Connection header received
0619 18:52:01.513964 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:01.513969 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:01.514051 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:01.514153 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:01.515035 (+ 882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:52:01.515040 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:52:01.515045 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:52:01.515061 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:52:01.515108 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:52:01.515126 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:52:01.515239 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:01.515286 (+ 47us) server_negotiation.cc:299] Negotiation successful
0619 18:52:01.515343 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 18:52:01.518730 24715 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c868847a3c9f4d65a1d21045159c1bf0: No bootstrap required, opened a new log
I20260619 18:52:01.526819 24715 raft_consensus.cc:348] T 00000000000000000000000000000000 P c868847a3c9f4d65a1d21045159c1bf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c868847a3c9f4d65a1d21045159c1bf0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34449 } }
I20260619 18:52:01.527316 24715 raft_consensus.cc:374] T 00000000000000000000000000000000 P c868847a3c9f4d65a1d21045159c1bf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:52:01.527428 24715 raft_consensus.cc:729] T 00000000000000000000000000000000 P c868847a3c9f4d65a1d21045159c1bf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c868847a3c9f4d65a1d21045159c1bf0, State: Initialized, Role: FOLLOWER
I20260619 18:52:01.528115 24715 consensus_queue.cc:260] T 00000000000000000000000000000000 P c868847a3c9f4d65a1d21045159c1bf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c868847a3c9f4d65a1d21045159c1bf0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34449 } }
I20260619 18:52:01.528427 24715 raft_consensus.cc:388] T 00000000000000000000000000000000 P c868847a3c9f4d65a1d21045159c1bf0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:52:01.528575 24715 raft_consensus.cc:482] T 00000000000000000000000000000000 P c868847a3c9f4d65a1d21045159c1bf0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:52:01.528751 24715 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c868847a3c9f4d65a1d21045159c1bf0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:52:01.531334 24715 raft_consensus.cc:504] T 00000000000000000000000000000000 P c868847a3c9f4d65a1d21045159c1bf0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c868847a3c9f4d65a1d21045159c1bf0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34449 } }
I20260619 18:52:01.532014 24715 leader_election.cc:302] T 00000000000000000000000000000000 P c868847a3c9f4d65a1d21045159c1bf0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c868847a3c9f4d65a1d21045159c1bf0; no voters:
I20260619 18:52:01.532825 24715 leader_election.cc:288] T 00000000000000000000000000000000 P c868847a3c9f4d65a1d21045159c1bf0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:52:01.533013 24725 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c868847a3c9f4d65a1d21045159c1bf0 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:52:01.533660 24725 raft_consensus.cc:686] T 00000000000000000000000000000000 P c868847a3c9f4d65a1d21045159c1bf0 [term 1 LEADER]: Becoming Leader. State: Replica: c868847a3c9f4d65a1d21045159c1bf0, State: Running, Role: LEADER
I20260619 18:52:01.534420 24725 consensus_queue.cc:237] T 00000000000000000000000000000000 P c868847a3c9f4d65a1d21045159c1bf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c868847a3c9f4d65a1d21045159c1bf0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34449 } }
I20260619 18:52:01.535686 24715 sys_catalog.cc:564] T 00000000000000000000000000000000 P c868847a3c9f4d65a1d21045159c1bf0 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:52:01.538080 24726 sys_catalog.cc:455] T 00000000000000000000000000000000 P c868847a3c9f4d65a1d21045159c1bf0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c868847a3c9f4d65a1d21045159c1bf0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c868847a3c9f4d65a1d21045159c1bf0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34449 } } }
I20260619 18:52:01.538628 24726 sys_catalog.cc:458] T 00000000000000000000000000000000 P c868847a3c9f4d65a1d21045159c1bf0 [sys.catalog]: This master's current role is: LEADER
I20260619 18:52:01.538439 24727 sys_catalog.cc:455] T 00000000000000000000000000000000 P c868847a3c9f4d65a1d21045159c1bf0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c868847a3c9f4d65a1d21045159c1bf0. Latest consensus state: current_term: 1 leader_uuid: "c868847a3c9f4d65a1d21045159c1bf0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c868847a3c9f4d65a1d21045159c1bf0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34449 } } }
I20260619 18:52:01.538776 24727 sys_catalog.cc:458] T 00000000000000000000000000000000 P c868847a3c9f4d65a1d21045159c1bf0 [sys.catalog]: This master's current role is: LEADER
I20260619 18:52:01.540904 24731 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:52:01.547888 24731 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:52:01.558527 24731 catalog_manager.cc:1347] Generated new cluster ID: e405872cd6a7407a8b8d4ee4f171a0f9
I20260619 18:52:01.558689 24731 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:52:01.573787 24731 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:52:01.574292 24731 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:52:01.585017 24731 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c868847a3c9f4d65a1d21045159c1bf0: Generated new TSK 0
I20260619 18:52:01.586124 24731 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:52:01.599174 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 24648
W20260619 18:52:01.600662 24710 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:33915
W20260619 18:52:01.601466 4045 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 24648: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 18:52:01.601637 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 24648
2026-06-19T18:52:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1932 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781894810715282-4045-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:52:01 dist-test-slave-m9wg krb5kdc[24746](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:52:01 dist-test-slave-m9wg krb5kdc[24746](info): set up 2 sockets
Jun 19 18:52:01 dist-test-slave-m9wg krb5kdc[24746](info): commencing operation
krb5kdc: starting...
W20260619 18:52:02.652909 4045 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.003s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781894810715282-4045-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781894810715282-4045-0/krb5kdc/test-user.keytab.
Jun 19 18:52:02 dist-test-slave-m9wg krb5kdc[24746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895122, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:52:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:52:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781894810715282-4045-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781894810715282-4045-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 18:52:02.959869 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781894810715282-4045-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781894810715282-4045-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781894810715282-4045-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39485
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781894810715282-4045-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39485
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:52:03.337545 24759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:52:03.337932 24759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:52:03.338080 24759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:52:03.346145 24759 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:52:03.346297 24759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:52:03.346383 24759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:52:03.346486 24759 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:52:03.346546 24759 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:52:03.357663 24759 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781894810715282-4045-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781894810715282-4045-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39485
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781894810715282-4045-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39485
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781894810715282-4045-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:52:03.359483 24759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:52:03.361213 24759 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:52:03.375288 24764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:52:03.375346 24765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:52:03.375255 24767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:52:03.377257 24759 server_base.cc:1034] running on GCE node
I20260619 18:52:03.378425 24759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:52:03.380587 24759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:52:03.381878 24759 hybrid_clock.cc:648] HybridClock initialized: now 1781895123381808 us; error 62 us; skew 500 ppm
I20260619 18:52:03.382359 24759 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:52:03 dist-test-slave-m9wg krb5kdc[24746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895123, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:52:03.386523 24759 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 18:52:03.389153 24759 webserver.cc:460] Webserver started at http://127.0.0.1:39487/ using document root <none> and password file <none>
I20260619 18:52:03.390148 24759 fs_manager.cc:359] Metadata directory not provided
I20260619 18:52:03.390321 24759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:52:03.390715 24759 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:52:03.394809 24759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781894810715282-4045-0/minicluster-data/master-0/data/instance:
uuid: "edc9d0fd769044a1b4055800f69dd3f5"
format_stamp: "Formatted at 2026-06-19 18:52:03 on dist-test-slave-m9wg"
server_key: "e8a6c9e2818bd7b1a684f1af33feb7fc"
server_key_iv: "78ad34fed77c6619c3e0221b4f3fc865"
server_key_version: "encryptionkey@0"
I20260619 18:52:03.395848 24759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781894810715282-4045-0/minicluster-data/master-0/wal/instance:
uuid: "edc9d0fd769044a1b4055800f69dd3f5"
format_stamp: "Formatted at 2026-06-19 18:52:03 on dist-test-slave-m9wg"
server_key: "e8a6c9e2818bd7b1a684f1af33feb7fc"
server_key_iv: "78ad34fed77c6619c3e0221b4f3fc865"
server_key_version: "encryptionkey@0"
I20260619 18:52:03.402969 24759 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260619 18:52:03.407251 24774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:52:03.408996 24759 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:52:03.409186 24759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781894810715282-4045-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781894810715282-4045-0/minicluster-data/master-0/wal
uuid: "edc9d0fd769044a1b4055800f69dd3f5"
format_stamp: "Formatted at 2026-06-19 18:52:03 on dist-test-slave-m9wg"
server_key: "e8a6c9e2818bd7b1a684f1af33feb7fc"
server_key_iv: "78ad34fed77c6619c3e0221b4f3fc865"
server_key_version: "encryptionkey@0"
I20260619 18:52:03.409406 24759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781894810715282-4045-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781894810715282-4045-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781894810715282-4045-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:52:03.429334 24759 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:52:03.434007 24759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:52:03.434283 24759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:52:03.453385 24759 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39485
I20260619 18:52:03.453406 24825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39485 every 8 connection(s)
I20260619 18:52:03.454922 24759 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
I20260619 18:52:03.459304 24826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:52:03.460201 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 24759
I20260619 18:52:03.460400 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781894810715282-4045-0/minicluster-data/master-0/wal/instance
I20260619 18:52:03.460891 4045 external_mini_cluster.cc:1428] Setting key c28ce3c8aba1fd9b8caedb8519d49dd6
I20260619 18:52:03.473891 24826 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edc9d0fd769044a1b4055800f69dd3f5: Bootstrap starting.
Jun 19 18:52:03 dist-test-slave-m9wg krb5kdc[24746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895122, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 18:52:03.478356 24826 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P edc9d0fd769044a1b4055800f69dd3f5: Neither blocks nor log segments found. Creating new log.
I20260619 18:52:03.479729 24826 log.cc:824] T 00000000000000000000000000000000 P edc9d0fd769044a1b4055800f69dd3f5: Log is configured to *not* fsync() on all Append() calls
I20260619 18:52:03.483671 24826 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edc9d0fd769044a1b4055800f69dd3f5: No bootstrap required, opened a new log
I20260619 18:52:03.486780 24829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:03.462817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49161 (local address 127.0.0.1:39485)
0619 18:52:03.463618 (+ 801us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:03.463630 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:03.463683 (+ 53us) server_negotiation.cc:407] Connection header received
0619 18:52:03.465669 (+ 1986us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:03.465695 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:03.466147 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:03.466524 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:03.468508 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:03.469900 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:03.471779 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:03.472567 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:03.472697 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:03.477894 (+ 5197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:52:03.477902 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:52:03.477920 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:52:03.477960 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:52:03.481563 (+ 3603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:03.482468 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:03.482506 (+ 38us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:03.482516 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:03.482768 (+ 252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:03.483270 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:03.483276 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:03.483279 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:03.484062 (+ 783us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:52:03.484412 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:03.484783 (+ 371us) server_negotiation.cc:299] Negotiation successful
0619 18:52:03.485258 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":344,"threads_started":1}
I20260619 18:52:03.490826 4045 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:52:03.493535 24826 raft_consensus.cc:348] T 00000000000000000000000000000000 P edc9d0fd769044a1b4055800f69dd3f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edc9d0fd769044a1b4055800f69dd3f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39485 } }
I20260619 18:52:03.494053 24826 raft_consensus.cc:374] T 00000000000000000000000000000000 P edc9d0fd769044a1b4055800f69dd3f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:52:03.494168 24826 raft_consensus.cc:729] T 00000000000000000000000000000000 P edc9d0fd769044a1b4055800f69dd3f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edc9d0fd769044a1b4055800f69dd3f5, State: Initialized, Role: FOLLOWER
I20260619 18:52:03.494939 24826 consensus_queue.cc:260] T 00000000000000000000000000000000 P edc9d0fd769044a1b4055800f69dd3f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edc9d0fd769044a1b4055800f69dd3f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39485 } }
I20260619 18:52:03.495280 24826 raft_consensus.cc:388] T 00000000000000000000000000000000 P edc9d0fd769044a1b4055800f69dd3f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:52:03.495430 24826 raft_consensus.cc:482] T 00000000000000000000000000000000 P edc9d0fd769044a1b4055800f69dd3f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:52:03.495605 24826 raft_consensus.cc:3037] T 00000000000000000000000000000000 P edc9d0fd769044a1b4055800f69dd3f5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:52:03.498525 24826 raft_consensus.cc:504] T 00000000000000000000000000000000 P edc9d0fd769044a1b4055800f69dd3f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edc9d0fd769044a1b4055800f69dd3f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39485 } }
I20260619 18:52:03.499086 24826 leader_election.cc:302] T 00000000000000000000000000000000 P edc9d0fd769044a1b4055800f69dd3f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: edc9d0fd769044a1b4055800f69dd3f5; no voters:
I20260619 18:52:03.499801 24826 leader_election.cc:288] T 00000000000000000000000000000000 P edc9d0fd769044a1b4055800f69dd3f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:52:03.499907 24836 raft_consensus.cc:2781] T 00000000000000000000000000000000 P edc9d0fd769044a1b4055800f69dd3f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:52:03.503536 24836 raft_consensus.cc:686] T 00000000000000000000000000000000 P edc9d0fd769044a1b4055800f69dd3f5 [term 1 LEADER]: Becoming Leader. State: Replica: edc9d0fd769044a1b4055800f69dd3f5, State: Running, Role: LEADER
I20260619 18:52:03.504463 24826 sys_catalog.cc:564] T 00000000000000000000000000000000 P edc9d0fd769044a1b4055800f69dd3f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:52:03.504348 24836 consensus_queue.cc:237] T 00000000000000000000000000000000 P edc9d0fd769044a1b4055800f69dd3f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edc9d0fd769044a1b4055800f69dd3f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39485 } }
I20260619 18:52:03.508585 24837 sys_catalog.cc:455] T 00000000000000000000000000000000 P edc9d0fd769044a1b4055800f69dd3f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "edc9d0fd769044a1b4055800f69dd3f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edc9d0fd769044a1b4055800f69dd3f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39485 } } }
I20260619 18:52:03.508615 24838 sys_catalog.cc:455] T 00000000000000000000000000000000 P edc9d0fd769044a1b4055800f69dd3f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader edc9d0fd769044a1b4055800f69dd3f5. Latest consensus state: current_term: 1 leader_uuid: "edc9d0fd769044a1b4055800f69dd3f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edc9d0fd769044a1b4055800f69dd3f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39485 } } }
I20260619 18:52:03.509116 24838 sys_catalog.cc:458] T 00000000000000000000000000000000 P edc9d0fd769044a1b4055800f69dd3f5 [sys.catalog]: This master's current role is: LEADER
I20260619 18:52:03.515023 24837 sys_catalog.cc:458] T 00000000000000000000000000000000 P edc9d0fd769044a1b4055800f69dd3f5 [sys.catalog]: This master's current role is: LEADER
I20260619 18:52:03.515146 24829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:03.496704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47019 (local address 127.0.0.1:39485)
0619 18:52:03.496909 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:03.496915 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:03.497288 (+ 373us) server_negotiation.cc:407] Connection header received
0619 18:52:03.497451 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:03.497456 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:03.497537 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:03.497650 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:03.499137 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:03.500332 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:03.502335 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:03.503051 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:03.503199 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:03.506307 (+ 3108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:52:03.506314 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:52:03.506321 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:52:03.506383 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:52:03.510494 (+ 4111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:03.511482 (+ 988us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:03.511488 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:03.511492 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:03.511657 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:03.512141 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:03.512176 (+ 35us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:03.512180 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:03.512654 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:52:03.514286 (+ 1632us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:03.514453 (+ 167us) server_negotiation.cc:299] Negotiation successful
0619 18:52:03.514731 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 18:52:03.516167 24845 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:52:03.522356 24845 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:52:03.533350 24845 catalog_manager.cc:1347] Generated new cluster ID: c47ccf7d858d4df78a5aee62780eed5b
I20260619 18:52:03.533495 24845 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:52:03.552709 24845 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:52:03.554247 24845 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:52:03.561246 24845 catalog_manager.cc:5868] T 00000000000000000000000000000000 P edc9d0fd769044a1b4055800f69dd3f5: Generated new TSK 0
I20260619 18:52:03.562327 24845 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:52:03.577242 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 24759
I20260619 18:52:03.578436 24821 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:52:03.682433 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 24759
2026-06-19T18:52:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2082 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781894810715282-4045-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:52:03 dist-test-slave-m9wg krb5kdc[24858](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:52:03 dist-test-slave-m9wg krb5kdc[24858](info): set up 2 sockets
Jun 19 18:52:03 dist-test-slave-m9wg krb5kdc[24858](info): commencing operation
krb5kdc: starting...
W20260619 18:52:04.724897 4045 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.002s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781894810715282-4045-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781894810715282-4045-0/krb5kdc/test-user.keytab.
Jun 19 18:52:04 dist-test-slave-m9wg krb5kdc[24858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:52:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:52:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781894810715282-4045-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781894810715282-4045-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 18:52:05.005640 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781894810715282-4045-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781894810715282-4045-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781894810715282-4045-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38765
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781894810715282-4045-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38765
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:52:05.392261 24871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:52:05.392661 24871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:52:05.392779 24871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:52:05.400890 24871 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:52:05.401042 24871 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:52:05.401108 24871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:52:05.401173 24871 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:52:05.401219 24871 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:52:05.412961 24871 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781894810715282-4045-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781894810715282-4045-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38765
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781894810715282-4045-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38765
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781894810715282-4045-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:52:05.414723 24871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:52:05.416822 24871 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:52:05.430255 24876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:52:05.430179 24877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:52:05.430225 24879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:52:05.431039 24871 server_base.cc:1034] running on GCE node
I20260619 18:52:05.431936 24871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:52:05.433651 24871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:52:05.434851 24871 hybrid_clock.cc:648] HybridClock initialized: now 1781895125434792 us; error 63 us; skew 500 ppm
I20260619 18:52:05.435217 24871 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:52:05 dist-test-slave-m9wg krb5kdc[24858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895125, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:52:05.438510 24871 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 18:52:05.440106 24871 webserver.cc:460] Webserver started at http://127.0.0.1:34129/ using document root <none> and password file <none>
I20260619 18:52:05.440902 24871 fs_manager.cc:359] Metadata directory not provided
I20260619 18:52:05.441002 24871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:52:05.441286 24871 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:52:05.444720 24871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781894810715282-4045-0/minicluster-data/master-0/data/instance:
uuid: "f21514b185df4be395f1a1902cf94d3d"
format_stamp: "Formatted at 2026-06-19 18:52:05 on dist-test-slave-m9wg"
server_key: "a76cbbc35e4c634dfc185754fc354a08"
server_key_iv: "fb8e8a8a09f837e5c6bf1bef2dc1f232"
server_key_version: "encryptionkey@0"
I20260619 18:52:05.445506 24871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781894810715282-4045-0/minicluster-data/master-0/wal/instance:
uuid: "f21514b185df4be395f1a1902cf94d3d"
format_stamp: "Formatted at 2026-06-19 18:52:05 on dist-test-slave-m9wg"
server_key: "a76cbbc35e4c634dfc185754fc354a08"
server_key_iv: "fb8e8a8a09f837e5c6bf1bef2dc1f232"
server_key_version: "encryptionkey@0"
I20260619 18:52:05.451277 24871 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260619 18:52:05.455035 24886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:52:05.456589 24871 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:52:05.456768 24871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781894810715282-4045-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781894810715282-4045-0/minicluster-data/master-0/wal
uuid: "f21514b185df4be395f1a1902cf94d3d"
format_stamp: "Formatted at 2026-06-19 18:52:05 on dist-test-slave-m9wg"
server_key: "a76cbbc35e4c634dfc185754fc354a08"
server_key_iv: "fb8e8a8a09f837e5c6bf1bef2dc1f232"
server_key_version: "encryptionkey@0"
I20260619 18:52:05.456957 24871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781894810715282-4045-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781894810715282-4045-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781894810715282-4045-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:52:05.472674 24871 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:52:05.477730 24871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:52:05.478067 24871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:52:05.498186 24871 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38765
I20260619 18:52:05.498210 24937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38765 every 8 connection(s)
I20260619 18:52:05.499924 24871 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
I20260619 18:52:05.504765 24938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:52:05.507438 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 24871
I20260619 18:52:05.507643 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781894810715282-4045-0/minicluster-data/master-0/wal/instance
I20260619 18:52:05.508181 4045 external_mini_cluster.cc:1428] Setting key 8d4691e974664967d6327d7ed61f6022
I20260619 18:52:05.519019 24938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f21514b185df4be395f1a1902cf94d3d: Bootstrap starting.
I20260619 18:52:05.523351 24938 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f21514b185df4be395f1a1902cf94d3d: Neither blocks nor log segments found. Creating new log.
I20260619 18:52:05.524019 24941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:05.510648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37063 (local address 127.0.0.1:38765)
0619 18:52:05.511644 (+ 996us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:05.511661 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:05.511724 (+ 63us) server_negotiation.cc:407] Connection header received
0619 18:52:05.513789 (+ 2065us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:05.513824 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:05.514408 (+ 584us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:05.514885 (+ 477us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:05.515581 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:05.517145 (+ 1564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:05.518636 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:05.519471 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:05.519666 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:05.521088 (+ 1422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:52:05.521097 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:52:05.521115 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:52:05.521795 (+ 680us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:52:05.521894 (+ 99us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:52:05.521921 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:52:05.522032 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:05.522297 (+ 265us) server_negotiation.cc:299] Negotiation successful
0619 18:52:05.522638 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":699,"thread_start_us":414,"threads_started":1}
I20260619 18:52:05.524974 24938 log.cc:824] T 00000000000000000000000000000000 P f21514b185df4be395f1a1902cf94d3d: Log is configured to *not* fsync() on all Append() calls
I20260619 18:52:05.528983 24938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f21514b185df4be395f1a1902cf94d3d: No bootstrap required, opened a new log
I20260619 18:52:05.529358 4045 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:52:05.542259 24938 raft_consensus.cc:348] T 00000000000000000000000000000000 P f21514b185df4be395f1a1902cf94d3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f21514b185df4be395f1a1902cf94d3d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38765 } }
I20260619 18:52:05.543037 24938 raft_consensus.cc:374] T 00000000000000000000000000000000 P f21514b185df4be395f1a1902cf94d3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:52:05.543226 24938 raft_consensus.cc:729] T 00000000000000000000000000000000 P f21514b185df4be395f1a1902cf94d3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21514b185df4be395f1a1902cf94d3d, State: Initialized, Role: FOLLOWER
I20260619 18:52:05.544102 24938 consensus_queue.cc:260] T 00000000000000000000000000000000 P f21514b185df4be395f1a1902cf94d3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f21514b185df4be395f1a1902cf94d3d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38765 } }
I20260619 18:52:05.544492 24938 raft_consensus.cc:388] T 00000000000000000000000000000000 P f21514b185df4be395f1a1902cf94d3d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:52:05.544664 24938 raft_consensus.cc:482] T 00000000000000000000000000000000 P f21514b185df4be395f1a1902cf94d3d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:52:05.544909 24938 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f21514b185df4be395f1a1902cf94d3d [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:52:05.545258 24941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:05.535017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41597 (local address 127.0.0.1:38765)
0619 18:52:05.535284 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:05.535291 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:05.535639 (+ 348us) server_negotiation.cc:407] Connection header received
0619 18:52:05.537652 (+ 2013us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:05.537670 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:05.537829 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:05.538081 (+ 252us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:05.539262 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:05.540614 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:05.542182 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:05.542871 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:05.543027 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:05.544373 (+ 1346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:52:05.544379 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:52:05.544384 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:52:05.544425 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:52:05.544494 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:52:05.544520 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:52:05.544624 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:05.544760 (+ 136us) server_negotiation.cc:299] Negotiation successful
0619 18:52:05.544889 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 18:52:05.548020 24938 raft_consensus.cc:504] T 00000000000000000000000000000000 P f21514b185df4be395f1a1902cf94d3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f21514b185df4be395f1a1902cf94d3d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38765 } }
I20260619 18:52:05.548684 24938 leader_election.cc:302] T 00000000000000000000000000000000 P f21514b185df4be395f1a1902cf94d3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f21514b185df4be395f1a1902cf94d3d; no voters:
I20260619 18:52:05.549316 24938 leader_election.cc:288] T 00000000000000000000000000000000 P f21514b185df4be395f1a1902cf94d3d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:52:05.549461 24948 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f21514b185df4be395f1a1902cf94d3d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:52:05.550156 24948 raft_consensus.cc:686] T 00000000000000000000000000000000 P f21514b185df4be395f1a1902cf94d3d [term 1 LEADER]: Becoming Leader. State: Replica: f21514b185df4be395f1a1902cf94d3d, State: Running, Role: LEADER
I20260619 18:52:05.550825 24948 consensus_queue.cc:237] T 00000000000000000000000000000000 P f21514b185df4be395f1a1902cf94d3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f21514b185df4be395f1a1902cf94d3d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38765 } }
I20260619 18:52:05.551846 24938 sys_catalog.cc:564] T 00000000000000000000000000000000 P f21514b185df4be395f1a1902cf94d3d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:52:05.554576 24949 sys_catalog.cc:455] T 00000000000000000000000000000000 P f21514b185df4be395f1a1902cf94d3d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f21514b185df4be395f1a1902cf94d3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f21514b185df4be395f1a1902cf94d3d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38765 } } }
I20260619 18:52:05.554543 24950 sys_catalog.cc:455] T 00000000000000000000000000000000 P f21514b185df4be395f1a1902cf94d3d [sys.catalog]: SysCatalogTable state changed. Reason: New leader f21514b185df4be395f1a1902cf94d3d. Latest consensus state: current_term: 1 leader_uuid: "f21514b185df4be395f1a1902cf94d3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f21514b185df4be395f1a1902cf94d3d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38765 } } }
I20260619 18:52:05.555023 24950 sys_catalog.cc:458] T 00000000000000000000000000000000 P f21514b185df4be395f1a1902cf94d3d [sys.catalog]: This master's current role is: LEADER
I20260619 18:52:05.555020 24949 sys_catalog.cc:458] T 00000000000000000000000000000000 P f21514b185df4be395f1a1902cf94d3d [sys.catalog]: This master's current role is: LEADER
I20260619 18:52:05.561077 24956 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:52:05.567154 24956 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:52:05.577683 24956 catalog_manager.cc:1347] Generated new cluster ID: 4e8b35e6cf284a2892dbe3306517c223
I20260619 18:52:05.577827 24956 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:52:05.601858 24956 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:52:05.603410 24956 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:52:05.611402 24956 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f21514b185df4be395f1a1902cf94d3d: Generated new TSK 0
I20260619 18:52:05.612494 24956 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:52:05.622373 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 24871
W20260619 18:52:05.623622 24933 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:37063
W20260619 18:52:05.624416 4045 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 24871: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 18:52:05.624572 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 24871
2026-06-19T18:52:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1941 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781894810715282-4045-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:52:05 dist-test-slave-m9wg krb5kdc[24969](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:52:05 dist-test-slave-m9wg krb5kdc[24969](info): set up 2 sockets
Jun 19 18:52:05 dist-test-slave-m9wg krb5kdc[24969](info): commencing operation
krb5kdc: starting...
W20260619 18:52:06.652750 4045 mini_kdc.cc:121] Time spent starting KDC: real 0.991s user 0.004s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781894810715282-4045-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781894810715282-4045-0/krb5kdc/test-user.keytab.
Jun 19 18:52:06 dist-test-slave-m9wg krb5kdc[24969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895126, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:52:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:52:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781894810715282-4045-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781894810715282-4045-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 18:52:06.945205 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781894810715282-4045-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781894810715282-4045-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781894810715282-4045-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44037
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781894810715282-4045-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44037
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:52:07.318219 24982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:52:07.318629 24982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:52:07.318732 24982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:52:07.326714 24982 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:52:07.326892 24982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:52:07.326967 24982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:52:07.327032 24982 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:52:07.327080 24982 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:52:07.338626 24982 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781894810715282-4045-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781894810715282-4045-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44037
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781894810715282-4045-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44037
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781894810715282-4045-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:52:07.340497 24982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:52:07.342228 24982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:52:07.356850 24987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:52:07.356804 24990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:52:07.356884 24988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:52:07.358006 24982 server_base.cc:1034] running on GCE node
I20260619 18:52:07.359090 24982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:52:07.361320 24982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:52:07.362593 24982 hybrid_clock.cc:648] HybridClock initialized: now 1781895127362536 us; error 58 us; skew 500 ppm
I20260619 18:52:07.363131 24982 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:52:07 dist-test-slave-m9wg krb5kdc[24969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895127, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:52:07.367441 24982 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 18:52:07.369549 24982 webserver.cc:460] Webserver started at http://127.0.0.1:39169/ using document root <none> and password file <none>
I20260619 18:52:07.370518 24982 fs_manager.cc:359] Metadata directory not provided
I20260619 18:52:07.370648 24982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:52:07.371033 24982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:52:07.375029 24982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781894810715282-4045-0/minicluster-data/master-0/data/instance:
uuid: "610c60c5ac564189a2f45d16f50fc1af"
format_stamp: "Formatted at 2026-06-19 18:52:07 on dist-test-slave-m9wg"
server_key: "e7f75c415690a446b92f35b5f909d21d"
server_key_iv: "afd18bfd0ae2a49419651ca72360515a"
server_key_version: "encryptionkey@0"
I20260619 18:52:07.376102 24982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781894810715282-4045-0/minicluster-data/master-0/wal/instance:
uuid: "610c60c5ac564189a2f45d16f50fc1af"
format_stamp: "Formatted at 2026-06-19 18:52:07 on dist-test-slave-m9wg"
server_key: "e7f75c415690a446b92f35b5f909d21d"
server_key_iv: "afd18bfd0ae2a49419651ca72360515a"
server_key_version: "encryptionkey@0"
I20260619 18:52:07.383733 24982 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260619 18:52:07.389451 24997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:52:07.391747 24982 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260619 18:52:07.392024 24982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781894810715282-4045-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781894810715282-4045-0/minicluster-data/master-0/wal
uuid: "610c60c5ac564189a2f45d16f50fc1af"
format_stamp: "Formatted at 2026-06-19 18:52:07 on dist-test-slave-m9wg"
server_key: "e7f75c415690a446b92f35b5f909d21d"
server_key_iv: "afd18bfd0ae2a49419651ca72360515a"
server_key_version: "encryptionkey@0"
I20260619 18:52:07.392436 24982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781894810715282-4045-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781894810715282-4045-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781894810715282-4045-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:52:07.400068 24982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:52:07.405364 24982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:52:07.405686 24982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:52:07.426170 24982 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44037
I20260619 18:52:07.426192 25048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44037 every 8 connection(s)
I20260619 18:52:07.428042 24982 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
I20260619 18:52:07.433315 25049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:52:07.434365 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 24982
I20260619 18:52:07.434532 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781894810715282-4045-0/minicluster-data/master-0/wal/instance
I20260619 18:52:07.435043 4045 external_mini_cluster.cc:1428] Setting key cddd766b7cba8e6c93051f9fd323f837
I20260619 18:52:07.445391 25052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:07.437402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35725 (local address 127.0.0.1:44037)
0619 18:52:07.438436 (+ 1034us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:07.438457 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:07.438518 (+ 61us) server_negotiation.cc:407] Connection header received
0619 18:52:07.440925 (+ 2407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:07.440963 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:07.441559 (+ 596us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:07.442110 (+ 551us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:07.443172 (+ 1062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:52:07.443181 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:52:07.443199 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:52:07.443217 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:52:07.443295 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:52:07.443321 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:52:07.443563 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:07.443701 (+ 138us) server_negotiation.cc:299] Negotiation successful
0619 18:52:07.443932 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":675,"thread_start_us":409,"threads_started":1}
I20260619 18:52:07.449221 25049 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 610c60c5ac564189a2f45d16f50fc1af: Bootstrap starting.
I20260619 18:52:07.449452 4045 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:52:07.452828 25049 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 610c60c5ac564189a2f45d16f50fc1af: Neither blocks nor log segments found. Creating new log.
I20260619 18:52:07.454015 25049 log.cc:824] T 00000000000000000000000000000000 P 610c60c5ac564189a2f45d16f50fc1af: Log is configured to *not* fsync() on all Append() calls
I20260619 18:52:07.456701 25052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:07.454237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59193 (local address 127.0.0.1:44037)
0619 18:52:07.454425 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:07.454430 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:07.454740 (+ 310us) server_negotiation.cc:407] Connection header received
0619 18:52:07.454893 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:07.454899 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:07.454965 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:07.455045 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:07.455982 (+ 937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:52:07.455988 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:52:07.455993 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:52:07.456011 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:52:07.456059 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:52:07.456075 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:52:07.456189 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:07.456367 (+ 178us) server_negotiation.cc:299] Negotiation successful
0619 18:52:07.456429 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 18:52:07.457484 25049 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 610c60c5ac564189a2f45d16f50fc1af: No bootstrap required, opened a new log
I20260619 18:52:07.465294 25049 raft_consensus.cc:348] T 00000000000000000000000000000000 P 610c60c5ac564189a2f45d16f50fc1af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "610c60c5ac564189a2f45d16f50fc1af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44037 } }
I20260619 18:52:07.465710 25049 raft_consensus.cc:374] T 00000000000000000000000000000000 P 610c60c5ac564189a2f45d16f50fc1af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:52:07.465804 25049 raft_consensus.cc:729] T 00000000000000000000000000000000 P 610c60c5ac564189a2f45d16f50fc1af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 610c60c5ac564189a2f45d16f50fc1af, State: Initialized, Role: FOLLOWER
I20260619 18:52:07.466424 25049 consensus_queue.cc:260] T 00000000000000000000000000000000 P 610c60c5ac564189a2f45d16f50fc1af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "610c60c5ac564189a2f45d16f50fc1af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44037 } }
I20260619 18:52:07.466671 25049 raft_consensus.cc:388] T 00000000000000000000000000000000 P 610c60c5ac564189a2f45d16f50fc1af [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:52:07.466795 25049 raft_consensus.cc:482] T 00000000000000000000000000000000 P 610c60c5ac564189a2f45d16f50fc1af [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:52:07.466954 25049 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 610c60c5ac564189a2f45d16f50fc1af [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:52:07.469558 25049 raft_consensus.cc:504] T 00000000000000000000000000000000 P 610c60c5ac564189a2f45d16f50fc1af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "610c60c5ac564189a2f45d16f50fc1af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44037 } }
I20260619 18:52:07.470103 25049 leader_election.cc:302] T 00000000000000000000000000000000 P 610c60c5ac564189a2f45d16f50fc1af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 610c60c5ac564189a2f45d16f50fc1af; no voters:
I20260619 18:52:07.470749 25049 leader_election.cc:288] T 00000000000000000000000000000000 P 610c60c5ac564189a2f45d16f50fc1af [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:52:07.470918 25059 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 610c60c5ac564189a2f45d16f50fc1af [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:52:07.471521 25059 raft_consensus.cc:686] T 00000000000000000000000000000000 P 610c60c5ac564189a2f45d16f50fc1af [term 1 LEADER]: Becoming Leader. State: Replica: 610c60c5ac564189a2f45d16f50fc1af, State: Running, Role: LEADER
I20260619 18:52:07.472193 25059 consensus_queue.cc:237] T 00000000000000000000000000000000 P 610c60c5ac564189a2f45d16f50fc1af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "610c60c5ac564189a2f45d16f50fc1af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44037 } }
I20260619 18:52:07.473097 25049 sys_catalog.cc:564] T 00000000000000000000000000000000 P 610c60c5ac564189a2f45d16f50fc1af [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:52:07.475204 25061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 610c60c5ac564189a2f45d16f50fc1af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 610c60c5ac564189a2f45d16f50fc1af. Latest consensus state: current_term: 1 leader_uuid: "610c60c5ac564189a2f45d16f50fc1af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "610c60c5ac564189a2f45d16f50fc1af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44037 } } }
I20260619 18:52:07.475654 25060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 610c60c5ac564189a2f45d16f50fc1af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "610c60c5ac564189a2f45d16f50fc1af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "610c60c5ac564189a2f45d16f50fc1af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44037 } } }
I20260619 18:52:07.475958 25061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 610c60c5ac564189a2f45d16f50fc1af [sys.catalog]: This master's current role is: LEADER
I20260619 18:52:07.475956 25060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 610c60c5ac564189a2f45d16f50fc1af [sys.catalog]: This master's current role is: LEADER
I20260619 18:52:07.477941 25067 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:52:07.484179 25067 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:52:07.494814 25067 catalog_manager.cc:1347] Generated new cluster ID: f7fd02282883426bb9f5419288c5d4e3
I20260619 18:52:07.494966 25067 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:52:07.537941 25067 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:52:07.538560 25067 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:52:07.553429 25067 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 610c60c5ac564189a2f45d16f50fc1af: Generated new TSK 0
I20260619 18:52:07.554639 25067 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:52:07.573684 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 24982
W20260619 18:52:07.575068 25044 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:35725
W20260619 18:52:07.575947 4045 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 24982: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 18:52:07.576098 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 24982
2026-06-19T18:52:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1951 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781894810715282-4045-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:52:07 dist-test-slave-m9wg krb5kdc[25080](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:52:07 dist-test-slave-m9wg krb5kdc[25080](info): set up 2 sockets
Jun 19 18:52:07 dist-test-slave-m9wg krb5kdc[25080](info): commencing operation
krb5kdc: starting...
W20260619 18:52:08.616827 4045 mini_kdc.cc:121] Time spent starting KDC: real 1.003s user 0.001s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781894810715282-4045-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781894810715282-4045-0/krb5kdc/test-user.keytab.
Jun 19 18:52:08 dist-test-slave-m9wg krb5kdc[25080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895128, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:52:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:52:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781894810715282-4045-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781894810715282-4045-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 18:52:08.921288 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781894810715282-4045-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781894810715282-4045-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781894810715282-4045-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38373
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781894810715282-4045-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38373
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:52:09.300828 25093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:52:09.301168 25093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:52:09.301321 25093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:52:09.308828 25093 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:52:09.308954 25093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:52:09.309037 25093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:52:09.309126 25093 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:52:09.309190 25093 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:52:09.319640 25093 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781894810715282-4045-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781894810715282-4045-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38373
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781894810715282-4045-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38373
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781894810715282-4045-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:52:09.321382 25093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:52:09.322992 25093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:52:09.335511 25098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:52:09.335583 25101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:52:09.335604 25099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:52:09.336416 25093 server_base.cc:1034] running on GCE node
I20260619 18:52:09.337234 25093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:52:09.338817 25093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:52:09.340029 25093 hybrid_clock.cc:648] HybridClock initialized: now 1781895129339983 us; error 38 us; skew 500 ppm
I20260619 18:52:09.340449 25093 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:52:09 dist-test-slave-m9wg krb5kdc[25080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895129, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:52:09.343787 25093 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 18:52:09.345419 25093 webserver.cc:460] Webserver started at http://127.0.0.1:46167/ using document root <none> and password file <none>
I20260619 18:52:09.346194 25093 fs_manager.cc:359] Metadata directory not provided
I20260619 18:52:09.346319 25093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:52:09.346640 25093 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:52:09.350347 25093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781894810715282-4045-0/minicluster-data/master-0/data/instance:
uuid: "d52ba2453b4d4d3497a11d44b419a83e"
format_stamp: "Formatted at 2026-06-19 18:52:09 on dist-test-slave-m9wg"
server_key: "611a4b105549d5d4373026b33c5135b0"
server_key_iv: "0d8b7023fe4116214cccbfea92bb9721"
server_key_version: "encryptionkey@0"
I20260619 18:52:09.351163 25093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781894810715282-4045-0/minicluster-data/master-0/wal/instance:
uuid: "d52ba2453b4d4d3497a11d44b419a83e"
format_stamp: "Formatted at 2026-06-19 18:52:09 on dist-test-slave-m9wg"
server_key: "611a4b105549d5d4373026b33c5135b0"
server_key_iv: "0d8b7023fe4116214cccbfea92bb9721"
server_key_version: "encryptionkey@0"
I20260619 18:52:09.357105 25093 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260619 18:52:09.361127 25108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:52:09.362672 25093 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 18:52:09.362848 25093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781894810715282-4045-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781894810715282-4045-0/minicluster-data/master-0/wal
uuid: "d52ba2453b4d4d3497a11d44b419a83e"
format_stamp: "Formatted at 2026-06-19 18:52:09 on dist-test-slave-m9wg"
server_key: "611a4b105549d5d4373026b33c5135b0"
server_key_iv: "0d8b7023fe4116214cccbfea92bb9721"
server_key_version: "encryptionkey@0"
I20260619 18:52:09.363056 25093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781894810715282-4045-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781894810715282-4045-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781894810715282-4045-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:52:09.377252 25093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:52:09.382056 25093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:52:09.382368 25093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:52:09.401407 25093 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38373
I20260619 18:52:09.401415 25159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38373 every 8 connection(s)
I20260619 18:52:09.403050 25093 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
I20260619 18:52:09.407655 25160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:52:09.409782 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 25093
I20260619 18:52:09.410000 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781894810715282-4045-0/minicluster-data/master-0/wal/instance
I20260619 18:52:09.410629 4045 external_mini_cluster.cc:1428] Setting key 4b30613a7f63fffe1d1a0c99167b1f9a
I20260619 18:52:09.423360 25160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d52ba2453b4d4d3497a11d44b419a83e: Bootstrap starting.
Jun 19 18:52:09 dist-test-slave-m9wg krb5kdc[25080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895128, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 18:52:09.427762 25160 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d52ba2453b4d4d3497a11d44b419a83e: Neither blocks nor log segments found. Creating new log.
I20260619 18:52:09.429037 25160 log.cc:824] T 00000000000000000000000000000000 P d52ba2453b4d4d3497a11d44b419a83e: Log is configured to *not* fsync() on all Append() calls
I20260619 18:52:09.432845 25160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d52ba2453b4d4d3497a11d44b419a83e: No bootstrap required, opened a new log
I20260619 18:52:09.436832 25163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:09.412761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.252:45937 (local address 127.0.0.1:38373)
0619 18:52:09.413637 (+ 876us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:09.413651 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:09.413705 (+ 54us) server_negotiation.cc:407] Connection header received
0619 18:52:09.415622 (+ 1917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:09.415654 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:09.416133 (+ 479us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:09.416579 (+ 446us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:09.418524 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:09.419931 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:09.421801 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:09.422512 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:09.422648 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:09.427730 (+ 5082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:52:09.427739 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:52:09.427758 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:52:09.427795 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:52:09.431517 (+ 3722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:09.432447 (+ 930us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:09.432457 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:09.432466 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:09.432682 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:09.433168 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:09.433174 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:09.433176 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:09.434006 (+ 830us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:52:09.434353 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:09.434739 (+ 386us) server_negotiation.cc:299] Negotiation successful
0619 18:52:09.435153 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":639,"thread_start_us":375,"threads_started":1}
I20260619 18:52:09.441346 4045 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:52:09.442885 25160 raft_consensus.cc:348] T 00000000000000000000000000000000 P d52ba2453b4d4d3497a11d44b419a83e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d52ba2453b4d4d3497a11d44b419a83e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38373 } }
I20260619 18:52:09.443399 25160 raft_consensus.cc:374] T 00000000000000000000000000000000 P d52ba2453b4d4d3497a11d44b419a83e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:52:09.443495 25160 raft_consensus.cc:729] T 00000000000000000000000000000000 P d52ba2453b4d4d3497a11d44b419a83e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d52ba2453b4d4d3497a11d44b419a83e, State: Initialized, Role: FOLLOWER
I20260619 18:52:09.444274 25160 consensus_queue.cc:260] T 00000000000000000000000000000000 P d52ba2453b4d4d3497a11d44b419a83e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d52ba2453b4d4d3497a11d44b419a83e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38373 } }
I20260619 18:52:09.444577 25160 raft_consensus.cc:388] T 00000000000000000000000000000000 P d52ba2453b4d4d3497a11d44b419a83e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:52:09.444725 25160 raft_consensus.cc:482] T 00000000000000000000000000000000 P d52ba2453b4d4d3497a11d44b419a83e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:52:09.444916 25160 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d52ba2453b4d4d3497a11d44b419a83e [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:52:09.448019 25160 raft_consensus.cc:504] T 00000000000000000000000000000000 P d52ba2453b4d4d3497a11d44b419a83e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d52ba2453b4d4d3497a11d44b419a83e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38373 } }
I20260619 18:52:09.448712 25160 leader_election.cc:302] T 00000000000000000000000000000000 P d52ba2453b4d4d3497a11d44b419a83e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d52ba2453b4d4d3497a11d44b419a83e; no voters:
I20260619 18:52:09.449364 25160 leader_election.cc:288] T 00000000000000000000000000000000 P d52ba2453b4d4d3497a11d44b419a83e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:52:09.449476 25170 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d52ba2453b4d4d3497a11d44b419a83e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:52:09.450577 25170 raft_consensus.cc:686] T 00000000000000000000000000000000 P d52ba2453b4d4d3497a11d44b419a83e [term 1 LEADER]: Becoming Leader. State: Replica: d52ba2453b4d4d3497a11d44b419a83e, State: Running, Role: LEADER
I20260619 18:52:09.451251 25170 consensus_queue.cc:237] T 00000000000000000000000000000000 P d52ba2453b4d4d3497a11d44b419a83e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d52ba2453b4d4d3497a11d44b419a83e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38373 } }
I20260619 18:52:09.452073 25160 sys_catalog.cc:564] T 00000000000000000000000000000000 P d52ba2453b4d4d3497a11d44b419a83e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:52:09.458122 25171 sys_catalog.cc:455] T 00000000000000000000000000000000 P d52ba2453b4d4d3497a11d44b419a83e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d52ba2453b4d4d3497a11d44b419a83e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d52ba2453b4d4d3497a11d44b419a83e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38373 } } }
I20260619 18:52:09.458541 25171 sys_catalog.cc:458] T 00000000000000000000000000000000 P d52ba2453b4d4d3497a11d44b419a83e [sys.catalog]: This master's current role is: LEADER
I20260619 18:52:09.459101 25172 sys_catalog.cc:455] T 00000000000000000000000000000000 P d52ba2453b4d4d3497a11d44b419a83e [sys.catalog]: SysCatalogTable state changed. Reason: New leader d52ba2453b4d4d3497a11d44b419a83e. Latest consensus state: current_term: 1 leader_uuid: "d52ba2453b4d4d3497a11d44b419a83e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d52ba2453b4d4d3497a11d44b419a83e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38373 } } }
I20260619 18:52:09.459398 25172 sys_catalog.cc:458] T 00000000000000000000000000000000 P d52ba2453b4d4d3497a11d44b419a83e [sys.catalog]: This master's current role is: LEADER
I20260619 18:52:09.463972 25163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:09.446966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.252:40429 (local address 127.0.0.1:38373)
0619 18:52:09.447160 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:09.447168 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:09.447714 (+ 546us) server_negotiation.cc:407] Connection header received
0619 18:52:09.447867 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:09.447872 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:09.447973 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:09.448092 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:09.449691 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:09.450911 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:09.452319 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:09.452783 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:09.452956 (+ 173us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:09.455586 (+ 2630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:52:09.455591 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:52:09.455595 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:52:09.455628 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:52:09.459344 (+ 3716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:09.460421 (+ 1077us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:09.460427 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:09.460431 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:09.460638 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:09.461209 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:09.461215 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:09.461219 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:09.461658 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:52:09.461941 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:09.463329 (+ 1388us) server_negotiation.cc:299] Negotiation successful
0619 18:52:09.463545 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":61}
I20260619 18:52:09.473333 25178 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:52:09.478492 25178 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:52:09.488085 25178 catalog_manager.cc:1347] Generated new cluster ID: a039eef2db824beca51fe41ce4746543
I20260619 18:52:09.488274 25178 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:52:09.500063 25178 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:52:09.501781 25178 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:52:09.508291 25178 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d52ba2453b4d4d3497a11d44b419a83e: Generated new TSK 0
I20260619 18:52:09.509282 25178 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:52:09.515383 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 25093
I20260619 18:52:09.516752 25155 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:52:09.615130 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 25093
W20260619 18:52:09.632365 25078 connection.cc:535] client connection to 127.0.0.1:38373 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38373): Connection reset by peer (error 104)
2026-06-19T18:52:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2039 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781894810715282-4045-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:52:09 dist-test-slave-m9wg krb5kdc[25193](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:52:09 dist-test-slave-m9wg krb5kdc[25193](info): set up 2 sockets
Jun 19 18:52:09 dist-test-slave-m9wg krb5kdc[25193](info): commencing operation
krb5kdc: starting...
W20260619 18:52:10.673005 4045 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.001s sys 0.036s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781894810715282-4045-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781894810715282-4045-0/krb5kdc/test-user.keytab.
Jun 19 18:52:10 dist-test-slave-m9wg krb5kdc[25193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895130, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:52:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:52:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781894810715282-4045-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781894810715282-4045-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 18:52:10.951321 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781894810715282-4045-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781894810715282-4045-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781894810715282-4045-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46679
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781894810715282-4045-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46679
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:52:11.336696 25206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:52:11.337031 25206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:52:11.337170 25206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:52:11.344597 25206 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:52:11.344714 25206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:52:11.344789 25206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:52:11.344882 25206 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:52:11.344944 25206 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:52:11.355198 25206 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781894810715282-4045-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781894810715282-4045-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46679
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781894810715282-4045-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781894810715282-4045-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:52:11.356771 25206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:52:11.358290 25206 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:52:11.370675 25211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:52:11.370632 25212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:52:11.370715 25214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:52:11.371752 25206 server_base.cc:1034] running on GCE node
I20260619 18:52:11.372702 25206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:52:11.374384 25206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:52:11.375591 25206 hybrid_clock.cc:648] HybridClock initialized: now 1781895131375545 us; error 43 us; skew 500 ppm
I20260619 18:52:11.375960 25206 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:52:11 dist-test-slave-m9wg krb5kdc[25193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895131, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:52:11.379405 25206 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 18:52:11.381130 25206 webserver.cc:460] Webserver started at http://127.0.0.1:44495/ using document root <none> and password file <none>
I20260619 18:52:11.381913 25206 fs_manager.cc:359] Metadata directory not provided
I20260619 18:52:11.382014 25206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:52:11.382303 25206 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:52:11.385722 25206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781894810715282-4045-0/minicluster-data/master-0/data/instance:
uuid: "a30dfa1c243549a39c1ce283928a5728"
format_stamp: "Formatted at 2026-06-19 18:52:11 on dist-test-slave-m9wg"
server_key: "70010ceb8e1197e4fe006d32ba2d15ac"
server_key_iv: "e1001a935ab79592f663c40871d25e3e"
server_key_version: "encryptionkey@0"
I20260619 18:52:11.386493 25206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781894810715282-4045-0/minicluster-data/master-0/wal/instance:
uuid: "a30dfa1c243549a39c1ce283928a5728"
format_stamp: "Formatted at 2026-06-19 18:52:11 on dist-test-slave-m9wg"
server_key: "70010ceb8e1197e4fe006d32ba2d15ac"
server_key_iv: "e1001a935ab79592f663c40871d25e3e"
server_key_version: "encryptionkey@0"
I20260619 18:52:11.392189 25206 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 18:52:11.395893 25221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:52:11.397374 25206 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 18:52:11.397536 25206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781894810715282-4045-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781894810715282-4045-0/minicluster-data/master-0/wal
uuid: "a30dfa1c243549a39c1ce283928a5728"
format_stamp: "Formatted at 2026-06-19 18:52:11 on dist-test-slave-m9wg"
server_key: "70010ceb8e1197e4fe006d32ba2d15ac"
server_key_iv: "e1001a935ab79592f663c40871d25e3e"
server_key_version: "encryptionkey@0"
I20260619 18:52:11.397733 25206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781894810715282-4045-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781894810715282-4045-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781894810715282-4045-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:52:11.432524 25206 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:52:11.437628 25206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:52:11.437954 25206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:52:11.458030 25206 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46679
I20260619 18:52:11.458030 25272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46679 every 8 connection(s)
I20260619 18:52:11.459884 25206 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
I20260619 18:52:11.461946 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 25206
I20260619 18:52:11.462157 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781894810715282-4045-0/minicluster-data/master-0/wal/instance
I20260619 18:52:11.462741 4045 external_mini_cluster.cc:1428] Setting key 5a2b26c1a43bbdced42a471890073f86
I20260619 18:52:11.465418 25273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:52:11.480014 25276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:11.465218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.252:58153 (local address 127.0.0.1:46679)
0619 18:52:11.466120 (+ 902us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:11.466139 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:11.466202 (+ 63us) server_negotiation.cc:407] Connection header received
0619 18:52:11.468721 (+ 2519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:11.468758 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:11.469438 (+ 680us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:11.469984 (+ 546us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:11.470592 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:11.472778 (+ 2186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:11.474216 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:11.475292 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:11.475486 (+ 194us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:11.476791 (+ 1305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:52:11.476807 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:52:11.476827 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:52:11.477713 (+ 886us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:52:11.477824 (+ 111us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:52:11.477861 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:52:11.478038 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:11.478230 (+ 192us) server_negotiation.cc:299] Negotiation successful
0619 18:52:11.478469 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":633,"thread_start_us":346,"threads_started":1}
I20260619 18:52:11.481274 25273 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a30dfa1c243549a39c1ce283928a5728: Bootstrap starting.
I20260619 18:52:11.484596 4045 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:52:11.485527 25273 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a30dfa1c243549a39c1ce283928a5728: Neither blocks nor log segments found. Creating new log.
I20260619 18:52:11.486886 25273 log.cc:824] T 00000000000000000000000000000000 P a30dfa1c243549a39c1ce283928a5728: Log is configured to *not* fsync() on all Append() calls
I20260619 18:52:11.490159 25273 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a30dfa1c243549a39c1ce283928a5728: No bootstrap required, opened a new log
I20260619 18:52:11.496804 25276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:11.489999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.252:60715 (local address 127.0.0.1:46679)
0619 18:52:11.490178 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:11.490183 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:11.490681 (+ 498us) server_negotiation.cc:407] Connection header received
0619 18:52:11.490837 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:11.490843 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:11.490925 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:11.491033 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:11.491542 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:11.492723 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:11.494051 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:11.494749 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:11.494875 (+ 126us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:11.495986 (+ 1111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:52:11.495992 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:52:11.495996 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:52:11.496018 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:52:11.496069 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:52:11.496089 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:52:11.496266 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:11.496401 (+ 135us) server_negotiation.cc:299] Negotiation successful
0619 18:52:11.496465 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 18:52:11.499485 25273 raft_consensus.cc:348] T 00000000000000000000000000000000 P a30dfa1c243549a39c1ce283928a5728 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a30dfa1c243549a39c1ce283928a5728" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46679 } }
I20260619 18:52:11.499859 25273 raft_consensus.cc:374] T 00000000000000000000000000000000 P a30dfa1c243549a39c1ce283928a5728 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:52:11.499934 25273 raft_consensus.cc:729] T 00000000000000000000000000000000 P a30dfa1c243549a39c1ce283928a5728 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a30dfa1c243549a39c1ce283928a5728, State: Initialized, Role: FOLLOWER
I20260619 18:52:11.500540 25273 consensus_queue.cc:260] T 00000000000000000000000000000000 P a30dfa1c243549a39c1ce283928a5728 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a30dfa1c243549a39c1ce283928a5728" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46679 } }
I20260619 18:52:11.500773 25273 raft_consensus.cc:388] T 00000000000000000000000000000000 P a30dfa1c243549a39c1ce283928a5728 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:52:11.500880 25273 raft_consensus.cc:482] T 00000000000000000000000000000000 P a30dfa1c243549a39c1ce283928a5728 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:52:11.501021 25273 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a30dfa1c243549a39c1ce283928a5728 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:52:11.503364 25273 raft_consensus.cc:504] T 00000000000000000000000000000000 P a30dfa1c243549a39c1ce283928a5728 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a30dfa1c243549a39c1ce283928a5728" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46679 } }
I20260619 18:52:11.503927 25273 leader_election.cc:302] T 00000000000000000000000000000000 P a30dfa1c243549a39c1ce283928a5728 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a30dfa1c243549a39c1ce283928a5728; no voters:
I20260619 18:52:11.504616 25273 leader_election.cc:288] T 00000000000000000000000000000000 P a30dfa1c243549a39c1ce283928a5728 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:52:11.504771 25283 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a30dfa1c243549a39c1ce283928a5728 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:52:11.505440 25283 raft_consensus.cc:686] T 00000000000000000000000000000000 P a30dfa1c243549a39c1ce283928a5728 [term 1 LEADER]: Becoming Leader. State: Replica: a30dfa1c243549a39c1ce283928a5728, State: Running, Role: LEADER
I20260619 18:52:11.506011 25283 consensus_queue.cc:237] T 00000000000000000000000000000000 P a30dfa1c243549a39c1ce283928a5728 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a30dfa1c243549a39c1ce283928a5728" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46679 } }
I20260619 18:52:11.506793 25273 sys_catalog.cc:564] T 00000000000000000000000000000000 P a30dfa1c243549a39c1ce283928a5728 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:52:11.509024 25285 sys_catalog.cc:455] T 00000000000000000000000000000000 P a30dfa1c243549a39c1ce283928a5728 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a30dfa1c243549a39c1ce283928a5728. Latest consensus state: current_term: 1 leader_uuid: "a30dfa1c243549a39c1ce283928a5728" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a30dfa1c243549a39c1ce283928a5728" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46679 } } }
I20260619 18:52:11.509009 25284 sys_catalog.cc:455] T 00000000000000000000000000000000 P a30dfa1c243549a39c1ce283928a5728 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a30dfa1c243549a39c1ce283928a5728" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a30dfa1c243549a39c1ce283928a5728" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46679 } } }
I20260619 18:52:11.509485 25285 sys_catalog.cc:458] T 00000000000000000000000000000000 P a30dfa1c243549a39c1ce283928a5728 [sys.catalog]: This master's current role is: LEADER
I20260619 18:52:11.509485 25284 sys_catalog.cc:458] T 00000000000000000000000000000000 P a30dfa1c243549a39c1ce283928a5728 [sys.catalog]: This master's current role is: LEADER
I20260619 18:52:11.515738 25289 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:52:11.521252 25289 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:52:11.530496 25289 catalog_manager.cc:1347] Generated new cluster ID: 3010c2b3b7bd46b1ac980cd03beeac8c
I20260619 18:52:11.530647 25289 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:52:11.569383 25289 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:52:11.571188 25289 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:52:11.583285 25289 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a30dfa1c243549a39c1ce283928a5728: Generated new TSK 0
I20260619 18:52:11.584738 25289 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:52:11.602305 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 25206
W20260619 18:52:11.603802 25268 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.252:58153
W20260619 18:52:11.604682 4045 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 25206: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 18:52:11.604836 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 25206
W20260619 18:52:11.621816 25191 connection.cc:535] client connection to 127.0.0.1:46679 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46679): Connection reset by peer (error 104)
2026-06-19T18:52:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1989 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781894810715282-4045-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:52:11 dist-test-slave-m9wg krb5kdc[25304](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:52:11 dist-test-slave-m9wg krb5kdc[25304](info): set up 2 sockets
Jun 19 18:52:11 dist-test-slave-m9wg krb5kdc[25304](info): commencing operation
krb5kdc: starting...
W20260619 18:52:12.664819 4045 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.042s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781894810715282-4045-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781894810715282-4045-0/krb5kdc/test-user.keytab.
Jun 19 18:52:12 dist-test-slave-m9wg krb5kdc[25304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895132, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:52:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:52:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781894810715282-4045-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781894810715282-4045-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 18:52:12.968559 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781894810715282-4045-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781894810715282-4045-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781894810715282-4045-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43679
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781894810715282-4045-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43679
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:52:13.350698 25317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:52:13.351089 25317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:52:13.351243 25317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:52:13.359225 25317 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:52:13.359402 25317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:52:13.359480 25317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:52:13.359573 25317 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:52:13.359642 25317 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:52:13.370971 25317 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781894810715282-4045-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781894810715282-4045-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43679
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781894810715282-4045-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781894810715282-4045-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:52:13.372768 25317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:52:13.374387 25317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:52:13.387053 25322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:52:13.387084 25323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:52:13.387005 25325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:52:13.387859 25317 server_base.cc:1034] running on GCE node
I20260619 18:52:13.388715 25317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:52:13.390250 25317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:52:13.391448 25317 hybrid_clock.cc:648] HybridClock initialized: now 1781895133391402 us; error 38 us; skew 500 ppm
I20260619 18:52:13.391803 25317 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:52:13 dist-test-slave-m9wg krb5kdc[25304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895133, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:52:13.400687 25317 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 18:52:13.402324 25317 webserver.cc:460] Webserver started at http://127.0.0.1:33869/ using document root <none> and password file <none>
I20260619 18:52:13.403126 25317 fs_manager.cc:359] Metadata directory not provided
I20260619 18:52:13.403249 25317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:52:13.403586 25317 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:52:13.407191 25317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781894810715282-4045-0/minicluster-data/master-0/data/instance:
uuid: "a15dec2972b644a9838e13b6a2d35b9c"
format_stamp: "Formatted at 2026-06-19 18:52:13 on dist-test-slave-m9wg"
server_key: "dd154204c0b2ba0761f80f63b67e80ad"
server_key_iv: "159871c710794fb7a005f639e0405254"
server_key_version: "encryptionkey@0"
I20260619 18:52:13.408034 25317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781894810715282-4045-0/minicluster-data/master-0/wal/instance:
uuid: "a15dec2972b644a9838e13b6a2d35b9c"
format_stamp: "Formatted at 2026-06-19 18:52:13 on dist-test-slave-m9wg"
server_key: "dd154204c0b2ba0761f80f63b67e80ad"
server_key_iv: "159871c710794fb7a005f639e0405254"
server_key_version: "encryptionkey@0"
I20260619 18:52:13.413842 25317 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260619 18:52:13.417655 25332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:52:13.419250 25317 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 18:52:13.419445 25317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781894810715282-4045-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781894810715282-4045-0/minicluster-data/master-0/wal
uuid: "a15dec2972b644a9838e13b6a2d35b9c"
format_stamp: "Formatted at 2026-06-19 18:52:13 on dist-test-slave-m9wg"
server_key: "dd154204c0b2ba0761f80f63b67e80ad"
server_key_iv: "159871c710794fb7a005f639e0405254"
server_key_version: "encryptionkey@0"
I20260619 18:52:13.419659 25317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781894810715282-4045-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781894810715282-4045-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781894810715282-4045-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:52:13.426524 25317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:52:13.431399 25317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:52:13.431702 25317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:52:13.451913 25317 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43679
I20260619 18:52:13.451938 25383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43679 every 8 connection(s)
I20260619 18:52:13.453652 25317 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
I20260619 18:52:13.458453 25384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:52:13.459520 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 25317
I20260619 18:52:13.459726 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781894810715282-4045-0/minicluster-data/master-0/wal/instance
I20260619 18:52:13.460449 4045 external_mini_cluster.cc:1428] Setting key f73f682eea98902d4bd225499c54aa87
I20260619 18:52:13.469358 25387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:13.462747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.252:40929 (local address 127.0.0.1:43679)
0619 18:52:13.463599 (+ 852us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:13.463616 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:13.463664 (+ 48us) server_negotiation.cc:407] Connection header received
0619 18:52:13.465595 (+ 1931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:13.465618 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:13.466037 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:13.466457 (+ 420us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:13.467587 (+ 1130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:52:13.467594 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:52:13.467608 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:52:13.467622 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:52:13.467685 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:52:13.467710 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:52:13.467934 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:13.468071 (+ 137us) server_negotiation.cc:299] Negotiation successful
0619 18:52:13.468319 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":354,"threads_started":1}
I20260619 18:52:13.473094 25384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a15dec2972b644a9838e13b6a2d35b9c: Bootstrap starting.
I20260619 18:52:13.473470 4045 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:52:13.476603 25384 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a15dec2972b644a9838e13b6a2d35b9c: Neither blocks nor log segments found. Creating new log.
I20260619 18:52:13.477767 25384 log.cc:824] T 00000000000000000000000000000000 P a15dec2972b644a9838e13b6a2d35b9c: Log is configured to *not* fsync() on all Append() calls
I20260619 18:52:13.480695 25387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:13.478052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.252:47833 (local address 127.0.0.1:43679)
0619 18:52:13.478193 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:13.478199 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:13.478863 (+ 664us) server_negotiation.cc:407] Connection header received
0619 18:52:13.479020 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:13.479024 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:13.479074 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:13.479166 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:13.480036 (+ 870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:52:13.480042 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:52:13.480047 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:52:13.480064 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:52:13.480111 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:52:13.480131 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:52:13.480308 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:13.480356 (+ 48us) server_negotiation.cc:299] Negotiation successful
0619 18:52:13.480419 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 18:52:13.481375 25384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a15dec2972b644a9838e13b6a2d35b9c: No bootstrap required, opened a new log
I20260619 18:52:13.489465 25384 raft_consensus.cc:348] T 00000000000000000000000000000000 P a15dec2972b644a9838e13b6a2d35b9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a15dec2972b644a9838e13b6a2d35b9c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43679 } }
I20260619 18:52:13.489853 25384 raft_consensus.cc:374] T 00000000000000000000000000000000 P a15dec2972b644a9838e13b6a2d35b9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:52:13.489956 25384 raft_consensus.cc:729] T 00000000000000000000000000000000 P a15dec2972b644a9838e13b6a2d35b9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a15dec2972b644a9838e13b6a2d35b9c, State: Initialized, Role: FOLLOWER
I20260619 18:52:13.490583 25384 consensus_queue.cc:260] T 00000000000000000000000000000000 P a15dec2972b644a9838e13b6a2d35b9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a15dec2972b644a9838e13b6a2d35b9c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43679 } }
I20260619 18:52:13.490881 25384 raft_consensus.cc:388] T 00000000000000000000000000000000 P a15dec2972b644a9838e13b6a2d35b9c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:52:13.491031 25384 raft_consensus.cc:482] T 00000000000000000000000000000000 P a15dec2972b644a9838e13b6a2d35b9c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:52:13.491221 25384 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a15dec2972b644a9838e13b6a2d35b9c [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:52:13.493546 25384 raft_consensus.cc:504] T 00000000000000000000000000000000 P a15dec2972b644a9838e13b6a2d35b9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a15dec2972b644a9838e13b6a2d35b9c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43679 } }
I20260619 18:52:13.494081 25384 leader_election.cc:302] T 00000000000000000000000000000000 P a15dec2972b644a9838e13b6a2d35b9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a15dec2972b644a9838e13b6a2d35b9c; no voters:
I20260619 18:52:13.494661 25384 leader_election.cc:288] T 00000000000000000000000000000000 P a15dec2972b644a9838e13b6a2d35b9c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:52:13.494782 25394 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a15dec2972b644a9838e13b6a2d35b9c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:52:13.495813 25394 raft_consensus.cc:686] T 00000000000000000000000000000000 P a15dec2972b644a9838e13b6a2d35b9c [term 1 LEADER]: Becoming Leader. State: Replica: a15dec2972b644a9838e13b6a2d35b9c, State: Running, Role: LEADER
I20260619 18:52:13.496380 25394 consensus_queue.cc:237] T 00000000000000000000000000000000 P a15dec2972b644a9838e13b6a2d35b9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a15dec2972b644a9838e13b6a2d35b9c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43679 } }
I20260619 18:52:13.496726 25384 sys_catalog.cc:564] T 00000000000000000000000000000000 P a15dec2972b644a9838e13b6a2d35b9c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:52:13.499878 25396 sys_catalog.cc:455] T 00000000000000000000000000000000 P a15dec2972b644a9838e13b6a2d35b9c [sys.catalog]: SysCatalogTable state changed. Reason: New leader a15dec2972b644a9838e13b6a2d35b9c. Latest consensus state: current_term: 1 leader_uuid: "a15dec2972b644a9838e13b6a2d35b9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a15dec2972b644a9838e13b6a2d35b9c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43679 } } }
I20260619 18:52:13.500473 25396 sys_catalog.cc:458] T 00000000000000000000000000000000 P a15dec2972b644a9838e13b6a2d35b9c [sys.catalog]: This master's current role is: LEADER
I20260619 18:52:13.500085 25395 sys_catalog.cc:455] T 00000000000000000000000000000000 P a15dec2972b644a9838e13b6a2d35b9c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a15dec2972b644a9838e13b6a2d35b9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a15dec2972b644a9838e13b6a2d35b9c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43679 } } }
I20260619 18:52:13.501045 25395 sys_catalog.cc:458] T 00000000000000000000000000000000 P a15dec2972b644a9838e13b6a2d35b9c [sys.catalog]: This master's current role is: LEADER
I20260619 18:52:13.501322 25403 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:52:13.507275 25403 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:52:13.517088 25403 catalog_manager.cc:1347] Generated new cluster ID: d82d4f65cece42ababc7b795bdd82d6b
I20260619 18:52:13.517191 25403 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:52:13.538910 25403 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:52:13.539398 25403 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:52:13.549358 25403 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a15dec2972b644a9838e13b6a2d35b9c: Generated new TSK 0
I20260619 18:52:13.550321 25403 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:52:13.565730 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 25317
W20260619 18:52:13.567631 25379 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.252:40929
W20260619 18:52:13.568367 4045 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 25317: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 18:52:13.568537 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 25317
2026-06-19T18:52:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1962 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781894810715282-4045-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:52:13 dist-test-slave-m9wg krb5kdc[25415](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:52:13 dist-test-slave-m9wg krb5kdc[25415](info): set up 2 sockets
Jun 19 18:52:13 dist-test-slave-m9wg krb5kdc[25415](info): commencing operation
krb5kdc: starting...
W20260619 18:52:14.612896 4045 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.005s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781894810715282-4045-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781894810715282-4045-0/krb5kdc/test-user.keytab.
Jun 19 18:52:14 dist-test-slave-m9wg krb5kdc[25415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895134, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:52:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:52:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781894810715282-4045-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781894810715282-4045-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 18:52:14.909487 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781894810715282-4045-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781894810715282-4045-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781894810715282-4045-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38121
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781894810715282-4045-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38121
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:52:15.284423 25428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:52:15.284911 25428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:52:15.285063 25428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:52:15.293964 25428 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:52:15.294112 25428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:52:15.294163 25428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:52:15.294239 25428 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:52:15.294286 25428 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:52:15.305299 25428 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781894810715282-4045-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781894810715282-4045-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38121
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781894810715282-4045-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781894810715282-4045-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:52:15.306898 25428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:52:15.308527 25428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:52:15.321465 25433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:52:15.321379 25436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:52:15.321422 25434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:52:15.322643 25428 server_base.cc:1034] running on GCE node
I20260619 18:52:15.323627 25428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:52:15.325341 25428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:52:15.326550 25428 hybrid_clock.cc:648] HybridClock initialized: now 1781895135326500 us; error 41 us; skew 500 ppm
I20260619 18:52:15.326929 25428 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:52:15 dist-test-slave-m9wg krb5kdc[25415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895135, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:52:15.330272 25428 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 18:52:15.331826 25428 webserver.cc:460] Webserver started at http://127.0.0.1:44613/ using document root <none> and password file <none>
I20260619 18:52:15.332659 25428 fs_manager.cc:359] Metadata directory not provided
I20260619 18:52:15.332777 25428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:52:15.333106 25428 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:52:15.336525 25428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781894810715282-4045-0/minicluster-data/master-0/data/instance:
uuid: "f03d0de1e08a476db06eb596458ebe5e"
format_stamp: "Formatted at 2026-06-19 18:52:15 on dist-test-slave-m9wg"
server_key: "052c03cab8f72df1aa9f0dc23a659d7f"
server_key_iv: "0e1609d848cc635a4725dbbba10e88df"
server_key_version: "encryptionkey@0"
I20260619 18:52:15.337355 25428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781894810715282-4045-0/minicluster-data/master-0/wal/instance:
uuid: "f03d0de1e08a476db06eb596458ebe5e"
format_stamp: "Formatted at 2026-06-19 18:52:15 on dist-test-slave-m9wg"
server_key: "052c03cab8f72df1aa9f0dc23a659d7f"
server_key_iv: "0e1609d848cc635a4725dbbba10e88df"
server_key_version: "encryptionkey@0"
I20260619 18:52:15.343176 25428 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260619 18:52:15.347360 25443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:52:15.349203 25428 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 18:52:15.349402 25428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781894810715282-4045-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781894810715282-4045-0/minicluster-data/master-0/wal
uuid: "f03d0de1e08a476db06eb596458ebe5e"
format_stamp: "Formatted at 2026-06-19 18:52:15 on dist-test-slave-m9wg"
server_key: "052c03cab8f72df1aa9f0dc23a659d7f"
server_key_iv: "0e1609d848cc635a4725dbbba10e88df"
server_key_version: "encryptionkey@0"
I20260619 18:52:15.349630 25428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781894810715282-4045-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781894810715282-4045-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781894810715282-4045-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:52:15.365942 25428 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:52:15.371166 25428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:52:15.371492 25428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:52:15.390735 25428 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38121
I20260619 18:52:15.390743 25494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38121 every 8 connection(s)
I20260619 18:52:15.392383 25428 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
I20260619 18:52:15.396912 25495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:52:15.399549 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 25428
I20260619 18:52:15.399750 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781894810715282-4045-0/minicluster-data/master-0/wal/instance
I20260619 18:52:15.400373 4045 external_mini_cluster.cc:1428] Setting key 2f0629e092dd07db80b527e8104fb755
I20260619 18:52:15.410497 25495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f03d0de1e08a476db06eb596458ebe5e: Bootstrap starting.
I20260619 18:52:15.414906 25495 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f03d0de1e08a476db06eb596458ebe5e: Neither blocks nor log segments found. Creating new log.
I20260619 18:52:15.416343 25495 log.cc:824] T 00000000000000000000000000000000 P f03d0de1e08a476db06eb596458ebe5e: Log is configured to *not* fsync() on all Append() calls
Jun 19 18:52:15 dist-test-slave-m9wg krb5kdc[25415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895134, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 18:52:15.419294 25495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f03d0de1e08a476db06eb596458ebe5e: No bootstrap required, opened a new log
I20260619 18:52:15.425158 25498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:15.402663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.252:56217 (local address 127.0.0.1:38121)
0619 18:52:15.403635 (+ 972us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:15.403652 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:15.403708 (+ 56us) server_negotiation.cc:407] Connection header received
0619 18:52:15.405621 (+ 1913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:15.405653 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:15.406148 (+ 495us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:15.406553 (+ 405us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:15.408442 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:15.409917 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:15.411702 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:15.412525 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:15.412678 (+ 153us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:15.417740 (+ 5062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:52:15.417749 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:52:15.417764 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:52:15.417806 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:52:15.420837 (+ 3031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:15.421595 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:15.421602 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:15.421609 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:15.421792 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:15.422250 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:15.422254 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:15.422256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:15.422823 (+ 567us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:52:15.423130 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:15.423656 (+ 526us) server_negotiation.cc:299] Negotiation successful
0619 18:52:15.423977 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":715,"thread_start_us":421,"threads_started":1}
I20260619 18:52:15.427698 25495 raft_consensus.cc:348] T 00000000000000000000000000000000 P f03d0de1e08a476db06eb596458ebe5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f03d0de1e08a476db06eb596458ebe5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38121 } }
I20260619 18:52:15.428120 25495 raft_consensus.cc:374] T 00000000000000000000000000000000 P f03d0de1e08a476db06eb596458ebe5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:52:15.428239 25495 raft_consensus.cc:729] T 00000000000000000000000000000000 P f03d0de1e08a476db06eb596458ebe5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f03d0de1e08a476db06eb596458ebe5e, State: Initialized, Role: FOLLOWER
I20260619 18:52:15.428941 25495 consensus_queue.cc:260] T 00000000000000000000000000000000 P f03d0de1e08a476db06eb596458ebe5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f03d0de1e08a476db06eb596458ebe5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38121 } }
I20260619 18:52:15.429301 25495 raft_consensus.cc:388] T 00000000000000000000000000000000 P f03d0de1e08a476db06eb596458ebe5e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:52:15.429451 25495 raft_consensus.cc:482] T 00000000000000000000000000000000 P f03d0de1e08a476db06eb596458ebe5e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:52:15.429487 4045 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:52:15.429617 25495 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f03d0de1e08a476db06eb596458ebe5e [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:52:15.431834 25495 raft_consensus.cc:504] T 00000000000000000000000000000000 P f03d0de1e08a476db06eb596458ebe5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f03d0de1e08a476db06eb596458ebe5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38121 } }
I20260619 18:52:15.432476 25495 leader_election.cc:302] T 00000000000000000000000000000000 P f03d0de1e08a476db06eb596458ebe5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f03d0de1e08a476db06eb596458ebe5e; no voters:
I20260619 18:52:15.433099 25495 leader_election.cc:288] T 00000000000000000000000000000000 P f03d0de1e08a476db06eb596458ebe5e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:52:15.433211 25504 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f03d0de1e08a476db06eb596458ebe5e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:52:15.435391 25504 raft_consensus.cc:686] T 00000000000000000000000000000000 P f03d0de1e08a476db06eb596458ebe5e [term 1 LEADER]: Becoming Leader. State: Replica: f03d0de1e08a476db06eb596458ebe5e, State: Running, Role: LEADER
I20260619 18:52:15.435660 25495 sys_catalog.cc:564] T 00000000000000000000000000000000 P f03d0de1e08a476db06eb596458ebe5e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:52:15.436096 25504 consensus_queue.cc:237] T 00000000000000000000000000000000 P f03d0de1e08a476db06eb596458ebe5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f03d0de1e08a476db06eb596458ebe5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38121 } }
I20260619 18:52:15.439570 25505 sys_catalog.cc:455] T 00000000000000000000000000000000 P f03d0de1e08a476db06eb596458ebe5e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f03d0de1e08a476db06eb596458ebe5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f03d0de1e08a476db06eb596458ebe5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38121 } } }
I20260619 18:52:15.440100 25506 sys_catalog.cc:455] T 00000000000000000000000000000000 P f03d0de1e08a476db06eb596458ebe5e [sys.catalog]: SysCatalogTable state changed. Reason: New leader f03d0de1e08a476db06eb596458ebe5e. Latest consensus state: current_term: 1 leader_uuid: "f03d0de1e08a476db06eb596458ebe5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f03d0de1e08a476db06eb596458ebe5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38121 } } }
I20260619 18:52:15.440491 25506 sys_catalog.cc:458] T 00000000000000000000000000000000 P f03d0de1e08a476db06eb596458ebe5e [sys.catalog]: This master's current role is: LEADER
I20260619 18:52:15.440928 25505 sys_catalog.cc:458] T 00000000000000000000000000000000 P f03d0de1e08a476db06eb596458ebe5e [sys.catalog]: This master's current role is: LEADER
I20260619 18:52:15.444557 25514 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:52:15.450992 25514 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:52:15.456228 25498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:15.434727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.252:51247 (local address 127.0.0.1:38121)
0619 18:52:15.434907 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:15.434913 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:15.434933 (+ 20us) server_negotiation.cc:407] Connection header received
0619 18:52:15.435121 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:15.435126 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:15.435235 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:15.435323 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:15.437136 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:15.438420 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:15.443290 (+ 4870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:15.443973 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:15.444118 (+ 145us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:15.447631 (+ 3513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:52:15.447642 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:52:15.447647 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:52:15.447691 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:52:15.451646 (+ 3955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:15.452820 (+ 1174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:15.452826 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:15.452830 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:15.452987 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:15.453614 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:15.453620 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:15.453623 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:15.454119 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:52:15.454465 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:15.455485 (+ 1020us) server_negotiation.cc:299] Negotiation successful
0619 18:52:15.455721 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 18:52:15.463578 25514 catalog_manager.cc:1347] Generated new cluster ID: b5826e9ef30c45158e49b797ce96f3a7
I20260619 18:52:15.463688 25514 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:52:15.476040 25514 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:52:15.477527 25514 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:52:15.492411 25514 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f03d0de1e08a476db06eb596458ebe5e: Generated new TSK 0
I20260619 18:52:15.493765 25514 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:52:15.507623 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 25428
I20260619 18:52:15.509382 25490 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:52:15.607960 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 25428
W20260619 18:52:15.625023 25413 connection.cc:535] client connection to 127.0.0.1:38121 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38121): Connection reset by peer (error 104)
2026-06-19T18:52:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2039 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781894810715282-4045-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:52:15 dist-test-slave-m9wg krb5kdc[25527](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:52:15 dist-test-slave-m9wg krb5kdc[25527](info): set up 2 sockets
Jun 19 18:52:15 dist-test-slave-m9wg krb5kdc[25527](info): commencing operation
krb5kdc: starting...
W20260619 18:52:16.652817 4045 mini_kdc.cc:121] Time spent starting KDC: real 1.004s user 0.003s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781894810715282-4045-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781894810715282-4045-0/krb5kdc/test-user.keytab.
Jun 19 18:52:16 dist-test-slave-m9wg krb5kdc[25527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895136, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:52:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:52:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781894810715282-4045-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781894810715282-4045-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 18:52:16.939707 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781894810715282-4045-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781894810715282-4045-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781894810715282-4045-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45065
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781894810715282-4045-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45065
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:52:17.320032 25540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:52:17.320441 25540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:52:17.320557 25540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:52:17.328130 25540 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:52:17.328289 25540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:52:17.328351 25540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:52:17.328414 25540 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:52:17.328459 25540 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:52:17.339015 25540 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781894810715282-4045-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781894810715282-4045-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45065
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781894810715282-4045-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45065
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781894810715282-4045-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:52:17.340698 25540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:52:17.342301 25540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:52:17.356099 25545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:52:17.356268 25548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:52:17.356251 25546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:52:17.357373 25540 server_base.cc:1034] running on GCE node
I20260619 18:52:17.358438 25540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:52:17.360453 25540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:52:17.361691 25540 hybrid_clock.cc:648] HybridClock initialized: now 1781895137361636 us; error 56 us; skew 500 ppm
I20260619 18:52:17.362125 25540 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:52:17 dist-test-slave-m9wg krb5kdc[25527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895137, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:52:17.366003 25540 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 18:52:17.367967 25540 webserver.cc:460] Webserver started at http://127.0.0.1:42245/ using document root <none> and password file <none>
I20260619 18:52:17.368921 25540 fs_manager.cc:359] Metadata directory not provided
I20260619 18:52:17.369058 25540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:52:17.369421 25540 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:52:17.373765 25540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781894810715282-4045-0/minicluster-data/master-0/data/instance:
uuid: "3e4a6c37807b437bbacc07557729222e"
format_stamp: "Formatted at 2026-06-19 18:52:17 on dist-test-slave-m9wg"
server_key: "b1a720e307ebf8eeb0aabdc771f802bd"
server_key_iv: "1da2ca86448ca2866e636db210685bec"
server_key_version: "encryptionkey@0"
I20260619 18:52:17.374866 25540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781894810715282-4045-0/minicluster-data/master-0/wal/instance:
uuid: "3e4a6c37807b437bbacc07557729222e"
format_stamp: "Formatted at 2026-06-19 18:52:17 on dist-test-slave-m9wg"
server_key: "b1a720e307ebf8eeb0aabdc771f802bd"
server_key_iv: "1da2ca86448ca2866e636db210685bec"
server_key_version: "encryptionkey@0"
I20260619 18:52:17.382042 25540 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.003s
I20260619 18:52:17.386690 25555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:52:17.388640 25540 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260619 18:52:17.388882 25540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781894810715282-4045-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781894810715282-4045-0/minicluster-data/master-0/wal
uuid: "3e4a6c37807b437bbacc07557729222e"
format_stamp: "Formatted at 2026-06-19 18:52:17 on dist-test-slave-m9wg"
server_key: "b1a720e307ebf8eeb0aabdc771f802bd"
server_key_iv: "1da2ca86448ca2866e636db210685bec"
server_key_version: "encryptionkey@0"
I20260619 18:52:17.389132 25540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781894810715282-4045-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781894810715282-4045-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781894810715282-4045-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:52:17.428849 25540 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:52:17.433955 25540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:52:17.434274 25540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:52:17.453903 25540 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45065
I20260619 18:52:17.453903 25606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45065 every 8 connection(s)
I20260619 18:52:17.455554 25540 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
I20260619 18:52:17.460083 25607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:52:17.460528 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 25540
I20260619 18:52:17.460696 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781894810715282-4045-0/minicluster-data/master-0/wal/instance
I20260619 18:52:17.461206 4045 external_mini_cluster.cc:1428] Setting key 9b8d0ac92dc1d2c49a8097ed5bd22897
I20260619 18:52:17.474702 25607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e4a6c37807b437bbacc07557729222e: Bootstrap starting.
I20260619 18:52:17.477310 25610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:17.463343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.252:49293 (local address 127.0.0.1:45065)
0619 18:52:17.464245 (+ 902us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:17.464258 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:17.464312 (+ 54us) server_negotiation.cc:407] Connection header received
0619 18:52:17.466615 (+ 2303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:17.466641 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:17.467130 (+ 489us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:17.467508 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:17.468115 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:17.469540 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:17.471483 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:17.472317 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:17.472475 (+ 158us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:17.473813 (+ 1338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:52:17.473822 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:52:17.473849 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:52:17.474486 (+ 637us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:52:17.474572 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:52:17.474604 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:52:17.474724 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:17.475415 (+ 691us) server_negotiation.cc:299] Negotiation successful
0619 18:52:17.475634 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":629,"thread_start_us":379,"threads_started":1}
I20260619 18:52:17.479199 25607 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e4a6c37807b437bbacc07557729222e: Neither blocks nor log segments found. Creating new log.
I20260619 18:52:17.480604 25607 log.cc:824] T 00000000000000000000000000000000 P 3e4a6c37807b437bbacc07557729222e: Log is configured to *not* fsync() on all Append() calls
I20260619 18:52:17.482069 4045 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:52:17.483942 25607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e4a6c37807b437bbacc07557729222e: No bootstrap required, opened a new log
I20260619 18:52:17.494331 25610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:17.486921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.252:37861 (local address 127.0.0.1:45065)
0619 18:52:17.487148 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:17.487155 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:17.487464 (+ 309us) server_negotiation.cc:407] Connection header received
0619 18:52:17.487677 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:17.487697 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:17.487772 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:17.487860 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:17.488551 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:17.489440 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:17.491372 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:17.491881 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:17.491982 (+ 101us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:17.493436 (+ 1454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:52:17.493442 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:52:17.493446 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:52:17.493470 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:52:17.493537 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:52:17.493554 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:52:17.493649 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:17.493863 (+ 214us) server_negotiation.cc:299] Negotiation successful
0619 18:52:17.493947 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260619 18:52:17.494256 25607 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e4a6c37807b437bbacc07557729222e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e4a6c37807b437bbacc07557729222e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45065 } }
I20260619 18:52:17.494896 25607 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e4a6c37807b437bbacc07557729222e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:52:17.495002 25607 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e4a6c37807b437bbacc07557729222e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e4a6c37807b437bbacc07557729222e, State: Initialized, Role: FOLLOWER
I20260619 18:52:17.495898 25607 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e4a6c37807b437bbacc07557729222e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e4a6c37807b437bbacc07557729222e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45065 } }
I20260619 18:52:17.496256 25607 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3e4a6c37807b437bbacc07557729222e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:52:17.496428 25607 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e4a6c37807b437bbacc07557729222e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:52:17.496606 25607 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e4a6c37807b437bbacc07557729222e [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:52:17.499842 25607 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e4a6c37807b437bbacc07557729222e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e4a6c37807b437bbacc07557729222e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45065 } }
I20260619 18:52:17.500612 25607 leader_election.cc:302] T 00000000000000000000000000000000 P 3e4a6c37807b437bbacc07557729222e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e4a6c37807b437bbacc07557729222e; no voters:
I20260619 18:52:17.501274 25607 leader_election.cc:288] T 00000000000000000000000000000000 P 3e4a6c37807b437bbacc07557729222e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:52:17.501437 25617 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e4a6c37807b437bbacc07557729222e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:52:17.502118 25617 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e4a6c37807b437bbacc07557729222e [term 1 LEADER]: Becoming Leader. State: Replica: 3e4a6c37807b437bbacc07557729222e, State: Running, Role: LEADER
I20260619 18:52:17.502732 25617 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e4a6c37807b437bbacc07557729222e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e4a6c37807b437bbacc07557729222e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45065 } }
I20260619 18:52:17.503692 25607 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e4a6c37807b437bbacc07557729222e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:52:17.505781 25618 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e4a6c37807b437bbacc07557729222e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3e4a6c37807b437bbacc07557729222e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e4a6c37807b437bbacc07557729222e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45065 } } }
I20260619 18:52:17.505808 25619 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e4a6c37807b437bbacc07557729222e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e4a6c37807b437bbacc07557729222e. Latest consensus state: current_term: 1 leader_uuid: "3e4a6c37807b437bbacc07557729222e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e4a6c37807b437bbacc07557729222e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45065 } } }
I20260619 18:52:17.506296 25618 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e4a6c37807b437bbacc07557729222e [sys.catalog]: This master's current role is: LEADER
I20260619 18:52:17.506318 25619 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e4a6c37807b437bbacc07557729222e [sys.catalog]: This master's current role is: LEADER
I20260619 18:52:17.507288 25624 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:52:17.515499 25624 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:52:17.525594 25624 catalog_manager.cc:1347] Generated new cluster ID: c82e306602fd49b896fe7a7bc6313b2c
I20260619 18:52:17.525746 25624 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:52:17.536074 25624 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:52:17.538076 25624 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:52:17.564257 25624 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3e4a6c37807b437bbacc07557729222e: Generated new TSK 0
I20260619 18:52:17.565728 25624 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:52:17.579669 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 25540
W20260619 18:52:17.581104 25602 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.252:49293
W20260619 18:52:17.581972 4045 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 25540: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 18:52:17.582149 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 25540
W20260619 18:52:17.599154 25525 connection.cc:535] client connection to 127.0.0.1:45065 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45065): Connection reset by peer (error 104)
2026-06-19T18:52:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1974 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781894810715282-4045-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:52:17 dist-test-slave-m9wg krb5kdc[25638](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:52:17 dist-test-slave-m9wg krb5kdc[25638](info): set up 2 sockets
Jun 19 18:52:17 dist-test-slave-m9wg krb5kdc[25638](info): commencing operation
krb5kdc: starting...
W20260619 18:52:18.624787 4045 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.011s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781894810715282-4045-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781894810715282-4045-0/krb5kdc/test-user.keytab.
Jun 19 18:52:18 dist-test-slave-m9wg krb5kdc[25638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895138, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:52:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:52:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781894810715282-4045-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781894810715282-4045-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 18:52:18.905051 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781894810715282-4045-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781894810715282-4045-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781894810715282-4045-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46629
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781894810715282-4045-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46629
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:52:19.283123 25651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:52:19.283551 25651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:52:19.283655 25651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:52:19.291764 25651 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:52:19.291916 25651 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:52:19.291965 25651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:52:19.292035 25651 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:52:19.292081 25651 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:52:19.303388 25651 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781894810715282-4045-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781894810715282-4045-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46629
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781894810715282-4045-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46629
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781894810715282-4045-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:52:19.305209 25651 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:52:19.306934 25651 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:52:19.321171 25656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:52:19.321218 25657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:52:19.321367 25659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:52:19.322111 25651 server_base.cc:1034] running on GCE node
I20260619 18:52:19.323105 25651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:52:19.325078 25651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:52:19.326316 25651 hybrid_clock.cc:648] HybridClock initialized: now 1781895139326249 us; error 57 us; skew 500 ppm
I20260619 18:52:19.326781 25651 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:52:19 dist-test-slave-m9wg krb5kdc[25638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895139, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:52:19.335170 25651 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 18:52:19.337472 25651 webserver.cc:460] Webserver started at http://127.0.0.1:35007/ using document root <none> and password file <none>
I20260619 18:52:19.338440 25651 fs_manager.cc:359] Metadata directory not provided
I20260619 18:52:19.338577 25651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:52:19.338949 25651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:52:19.342819 25651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781894810715282-4045-0/minicluster-data/master-0/data/instance:
uuid: "25ec6190a7274304953deed9e3a58e3d"
format_stamp: "Formatted at 2026-06-19 18:52:19 on dist-test-slave-m9wg"
server_key: "b337de13d973b4eae9166de4c44c8951"
server_key_iv: "b0a64a0721cb9f3025bd439d66cfa69f"
server_key_version: "encryptionkey@0"
I20260619 18:52:19.343688 25651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781894810715282-4045-0/minicluster-data/master-0/wal/instance:
uuid: "25ec6190a7274304953deed9e3a58e3d"
format_stamp: "Formatted at 2026-06-19 18:52:19 on dist-test-slave-m9wg"
server_key: "b337de13d973b4eae9166de4c44c8951"
server_key_iv: "b0a64a0721cb9f3025bd439d66cfa69f"
server_key_version: "encryptionkey@0"
I20260619 18:52:19.349862 25651 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260619 18:52:19.353678 25666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:52:19.355190 25651 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 18:52:19.355383 25651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781894810715282-4045-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781894810715282-4045-0/minicluster-data/master-0/wal
uuid: "25ec6190a7274304953deed9e3a58e3d"
format_stamp: "Formatted at 2026-06-19 18:52:19 on dist-test-slave-m9wg"
server_key: "b337de13d973b4eae9166de4c44c8951"
server_key_iv: "b0a64a0721cb9f3025bd439d66cfa69f"
server_key_version: "encryptionkey@0"
I20260619 18:52:19.355576 25651 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781894810715282-4045-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781894810715282-4045-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781894810715282-4045-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:52:19.361804 25651 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:52:19.366078 25651 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:52:19.366360 25651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:52:19.384905 25651 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46629
I20260619 18:52:19.384953 25717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46629 every 8 connection(s)
I20260619 18:52:19.386468 25651 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
I20260619 18:52:19.390799 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 25651
I20260619 18:52:19.390877 25718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:52:19.390998 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781894810715282-4045-0/minicluster-data/master-0/wal/instance
I20260619 18:52:19.391523 4045 external_mini_cluster.cc:1428] Setting key 991df439f3599ec0c33c47ceee66a37b
I20260619 18:52:19.400512 25721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:19.393512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.252:59545 (local address 127.0.0.1:46629)
0619 18:52:19.394427 (+ 915us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:19.394443 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:19.394505 (+ 62us) server_negotiation.cc:407] Connection header received
0619 18:52:19.396299 (+ 1794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:19.396333 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:19.396850 (+ 517us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:19.397348 (+ 498us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:19.398414 (+ 1066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:52:19.398423 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:52:19.398441 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:52:19.398459 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:52:19.398534 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:52:19.398567 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:52:19.398807 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:19.398954 (+ 147us) server_negotiation.cc:299] Negotiation successful
0619 18:52:19.399169 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":637,"thread_start_us":357,"threads_started":1}
I20260619 18:52:19.404247 4045 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:52:19.406064 25718 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25ec6190a7274304953deed9e3a58e3d: Bootstrap starting.
I20260619 18:52:19.409755 25718 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 25ec6190a7274304953deed9e3a58e3d: Neither blocks nor log segments found. Creating new log.
I20260619 18:52:19.411036 25718 log.cc:824] T 00000000000000000000000000000000 P 25ec6190a7274304953deed9e3a58e3d: Log is configured to *not* fsync() on all Append() calls
I20260619 18:52:19.411258 25721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:19.408880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.252:42297 (local address 127.0.0.1:46629)
0619 18:52:19.409046 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:19.409051 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:19.409372 (+ 321us) server_negotiation.cc:407] Connection header received
0619 18:52:19.409508 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:19.409514 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:19.409574 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:19.409683 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:19.410598 (+ 915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:52:19.410603 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:52:19.410608 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:52:19.410626 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:52:19.410680 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:52:19.410700 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:52:19.410830 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:19.410877 (+ 47us) server_negotiation.cc:299] Negotiation successful
0619 18:52:19.410935 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 18:52:19.414196 25718 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25ec6190a7274304953deed9e3a58e3d: No bootstrap required, opened a new log
I20260619 18:52:19.422019 25718 raft_consensus.cc:348] T 00000000000000000000000000000000 P 25ec6190a7274304953deed9e3a58e3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25ec6190a7274304953deed9e3a58e3d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46629 } }
I20260619 18:52:19.422397 25718 raft_consensus.cc:374] T 00000000000000000000000000000000 P 25ec6190a7274304953deed9e3a58e3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:52:19.422477 25718 raft_consensus.cc:729] T 00000000000000000000000000000000 P 25ec6190a7274304953deed9e3a58e3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25ec6190a7274304953deed9e3a58e3d, State: Initialized, Role: FOLLOWER
I20260619 18:52:19.423060 25718 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25ec6190a7274304953deed9e3a58e3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25ec6190a7274304953deed9e3a58e3d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46629 } }
I20260619 18:52:19.423292 25718 raft_consensus.cc:388] T 00000000000000000000000000000000 P 25ec6190a7274304953deed9e3a58e3d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:52:19.423398 25718 raft_consensus.cc:482] T 00000000000000000000000000000000 P 25ec6190a7274304953deed9e3a58e3d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:52:19.423529 25718 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25ec6190a7274304953deed9e3a58e3d [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:52:19.425983 25718 raft_consensus.cc:504] T 00000000000000000000000000000000 P 25ec6190a7274304953deed9e3a58e3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25ec6190a7274304953deed9e3a58e3d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46629 } }
I20260619 18:52:19.426496 25718 leader_election.cc:302] T 00000000000000000000000000000000 P 25ec6190a7274304953deed9e3a58e3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 25ec6190a7274304953deed9e3a58e3d; no voters:
I20260619 18:52:19.427042 25718 leader_election.cc:288] T 00000000000000000000000000000000 P 25ec6190a7274304953deed9e3a58e3d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:52:19.427171 25728 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 25ec6190a7274304953deed9e3a58e3d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:52:19.427882 25728 raft_consensus.cc:686] T 00000000000000000000000000000000 P 25ec6190a7274304953deed9e3a58e3d [term 1 LEADER]: Becoming Leader. State: Replica: 25ec6190a7274304953deed9e3a58e3d, State: Running, Role: LEADER
I20260619 18:52:19.428437 25728 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25ec6190a7274304953deed9e3a58e3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25ec6190a7274304953deed9e3a58e3d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46629 } }
I20260619 18:52:19.429235 25718 sys_catalog.cc:564] T 00000000000000000000000000000000 P 25ec6190a7274304953deed9e3a58e3d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:52:19.431452 25730 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25ec6190a7274304953deed9e3a58e3d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25ec6190a7274304953deed9e3a58e3d. Latest consensus state: current_term: 1 leader_uuid: "25ec6190a7274304953deed9e3a58e3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25ec6190a7274304953deed9e3a58e3d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46629 } } }
I20260619 18:52:19.431670 25729 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25ec6190a7274304953deed9e3a58e3d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "25ec6190a7274304953deed9e3a58e3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25ec6190a7274304953deed9e3a58e3d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46629 } } }
I20260619 18:52:19.431915 25730 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25ec6190a7274304953deed9e3a58e3d [sys.catalog]: This master's current role is: LEADER
I20260619 18:52:19.432054 25729 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25ec6190a7274304953deed9e3a58e3d [sys.catalog]: This master's current role is: LEADER
I20260619 18:52:19.438179 25734 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:52:19.444231 25734 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:52:19.445034 25745 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 18:52:19.454496 25734 catalog_manager.cc:1347] Generated new cluster ID: b50b1116b30a4fdc8128581c1e4f782e
I20260619 18:52:19.454610 25734 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:52:19.492102 25734 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:52:19.492614 25734 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:52:19.506665 25734 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 25ec6190a7274304953deed9e3a58e3d: Generated new TSK 0
I20260619 18:52:19.507675 25734 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:52:19.518103 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 25651
W20260619 18:52:19.519268 25713 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.252:59545
W20260619 18:52:19.520169 4045 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 25651: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 18:52:19.520320 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 25651
2026-06-19T18:52:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1938 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781894810715282-4045-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:52:19 dist-test-slave-m9wg krb5kdc[25749](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:52:19 dist-test-slave-m9wg krb5kdc[25749](info): set up 2 sockets
Jun 19 18:52:19 dist-test-slave-m9wg krb5kdc[25749](info): commencing operation
krb5kdc: starting...
W20260619 18:52:20.572855 4045 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.040s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781894810715282-4045-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781894810715282-4045-0/krb5kdc/test-user.keytab.
Jun 19 18:52:20 dist-test-slave-m9wg krb5kdc[25749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895140, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:52:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:52:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.243.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab.
Entry for principal kudu/127.3.243.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab.
I20260619 18:52:20.864642 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781894810715282-4045-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781894810715282-4045-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781894810715282-4045-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.126:34043
--webserver_interface=127.3.243.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab
--principal=kudu/127.3.243.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:41243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.126:34043
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:52:21.245299 25762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:52:21.245664 25762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:52:21.245808 25762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:52:21.253358 25762 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:52:21.253489 25762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:52:21.253568 25762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:52:21.253654 25762 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:52:21.253715 25762 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:52:21.264190 25762 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:41243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781894810715282-4045-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781894810715282-4045-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.243.126:34043
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab
--principal=kudu/127.3.243.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.126:34043
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.243.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781894810715282-4045-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:52:21.265770 25762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:52:21.267277 25762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:52:21.280211 25767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:52:21.280118 25770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:52:21.280175 25768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:52:21.280992 25762 server_base.cc:1034] running on GCE node
I20260619 18:52:21.281798 25762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:52:21.283317 25762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:52:21.284508 25762 hybrid_clock.cc:648] HybridClock initialized: now 1781895141284458 us; error 47 us; skew 500 ppm
I20260619 18:52:21.284852 25762 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:52:21 dist-test-slave-m9wg krb5kdc[25749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895141, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:52:21.288128 25762 init.cc:377] Logged in from keytab as kudu/127.3.243.126@KRBTEST.COM (short username kudu)
I20260619 18:52:21.289716 25762 webserver.cc:460] Webserver started at http://127.3.243.126:33195/ using document root <none> and password file <none>
I20260619 18:52:21.290443 25762 fs_manager.cc:359] Metadata directory not provided
I20260619 18:52:21.290561 25762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:52:21.290868 25762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:52:21.294178 25762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781894810715282-4045-0/minicluster-data/master-0/data/instance:
uuid: "12c220661554431991865b1220e76e16"
format_stamp: "Formatted at 2026-06-19 18:52:21 on dist-test-slave-m9wg"
server_key: "a4cf9e5468fde252e2f93276e756149f"
server_key_iv: "80d3e5d2b4312213768ba56599e850a5"
server_key_version: "encryptionkey@0"
I20260619 18:52:21.294996 25762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781894810715282-4045-0/minicluster-data/master-0/wal/instance:
uuid: "12c220661554431991865b1220e76e16"
format_stamp: "Formatted at 2026-06-19 18:52:21 on dist-test-slave-m9wg"
server_key: "a4cf9e5468fde252e2f93276e756149f"
server_key_iv: "80d3e5d2b4312213768ba56599e850a5"
server_key_version: "encryptionkey@0"
I20260619 18:52:21.300784 25762 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260619 18:52:21.304893 25777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:52:21.306663 25762 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 18:52:21.306872 25762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781894810715282-4045-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781894810715282-4045-0/minicluster-data/master-0/wal
uuid: "12c220661554431991865b1220e76e16"
format_stamp: "Formatted at 2026-06-19 18:52:21 on dist-test-slave-m9wg"
server_key: "a4cf9e5468fde252e2f93276e756149f"
server_key_iv: "80d3e5d2b4312213768ba56599e850a5"
server_key_version: "encryptionkey@0"
I20260619 18:52:21.307102 25762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781894810715282-4045-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781894810715282-4045-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781894810715282-4045-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:52:21.338444 25762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:52:21.343341 25762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:52:21.343629 25762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:52:21.362304 25762 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.126:34043
I20260619 18:52:21.362314 25828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.126:34043 every 8 connection(s)
I20260619 18:52:21.364022 25762 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
I20260619 18:52:21.368459 25829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:52:21.373260 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 25762
I20260619 18:52:21.373437 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781894810715282-4045-0/minicluster-data/master-0/wal/instance
I20260619 18:52:21.373941 4045 external_mini_cluster.cc:1428] Setting key 8ee5b47e42d7c878c8d3185ccd7c3eb5
I20260619 18:52:21.380599 25829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12c220661554431991865b1220e76e16: Bootstrap starting.
I20260619 18:52:21.384596 25829 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12c220661554431991865b1220e76e16: Neither blocks nor log segments found. Creating new log.
I20260619 18:52:21.385875 25829 log.cc:824] T 00000000000000000000000000000000 P 12c220661554431991865b1220e76e16: Log is configured to *not* fsync() on all Append() calls
Jun 19 18:52:21 dist-test-slave-m9wg krb5kdc[25749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895140, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.126@KRBTEST.COM
I20260619 18:52:21.389343 25829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12c220661554431991865b1220e76e16: No bootstrap required, opened a new log
I20260619 18:52:21.399353 25832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:21.375820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32783 (local address 127.3.243.126:34043)
0619 18:52:21.376625 (+ 805us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:21.376638 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:21.376687 (+ 49us) server_negotiation.cc:407] Connection header received
0619 18:52:21.378894 (+ 2207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:21.378919 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:21.379386 (+ 467us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:21.379725 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:21.381672 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:21.383046 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:21.384856 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:21.385572 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:21.385702 (+ 130us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:21.390391 (+ 4689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:52:21.390403 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:52:21.390422 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:52:21.390478 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:52:21.394478 (+ 4000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:21.395273 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:21.395282 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:21.395291 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:21.395507 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:21.395912 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:21.395917 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:21.395920 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:21.396704 (+ 784us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:52:21.397079 (+ 375us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:21.397406 (+ 327us) server_negotiation.cc:299] Negotiation successful
0619 18:52:21.397800 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":389,"threads_started":1}
I20260619 18:52:21.400110 25829 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12c220661554431991865b1220e76e16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12c220661554431991865b1220e76e16" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 34043 } }
I20260619 18:52:21.400553 25829 raft_consensus.cc:374] T 00000000000000000000000000000000 P 12c220661554431991865b1220e76e16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:52:21.400663 25829 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12c220661554431991865b1220e76e16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12c220661554431991865b1220e76e16, State: Initialized, Role: FOLLOWER
I20260619 18:52:21.401428 25829 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12c220661554431991865b1220e76e16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12c220661554431991865b1220e76e16" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 34043 } }
I20260619 18:52:21.401757 25829 raft_consensus.cc:388] T 00000000000000000000000000000000 P 12c220661554431991865b1220e76e16 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:52:21.401901 25829 raft_consensus.cc:482] T 00000000000000000000000000000000 P 12c220661554431991865b1220e76e16 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:52:21.402107 25829 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12c220661554431991865b1220e76e16 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:52:21.404304 4045 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:52:21.405457 25829 raft_consensus.cc:504] T 00000000000000000000000000000000 P 12c220661554431991865b1220e76e16 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12c220661554431991865b1220e76e16" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 34043 } }
I20260619 18:52:21.406194 25829 leader_election.cc:302] T 00000000000000000000000000000000 P 12c220661554431991865b1220e76e16 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 12c220661554431991865b1220e76e16; no voters:
I20260619 18:52:21.406957 25829 leader_election.cc:288] T 00000000000000000000000000000000 P 12c220661554431991865b1220e76e16 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:52:21.407051 25834 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 12c220661554431991865b1220e76e16 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:52:21.408254 25834 raft_consensus.cc:686] T 00000000000000000000000000000000 P 12c220661554431991865b1220e76e16 [term 1 LEADER]: Becoming Leader. State: Replica: 12c220661554431991865b1220e76e16, State: Running, Role: LEADER
I20260619 18:52:21.408996 25834 consensus_queue.cc:237] T 00000000000000000000000000000000 P 12c220661554431991865b1220e76e16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12c220661554431991865b1220e76e16" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 34043 } }
I20260619 18:52:21.409833 25829 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12c220661554431991865b1220e76e16 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:52:21.412582 25836 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12c220661554431991865b1220e76e16 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12c220661554431991865b1220e76e16. Latest consensus state: current_term: 1 leader_uuid: "12c220661554431991865b1220e76e16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12c220661554431991865b1220e76e16" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 34043 } } }
I20260619 18:52:21.412979 25836 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12c220661554431991865b1220e76e16 [sys.catalog]: This master's current role is: LEADER
I20260619 18:52:21.413533 25835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12c220661554431991865b1220e76e16 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "12c220661554431991865b1220e76e16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12c220661554431991865b1220e76e16" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 34043 } } }
I20260619 18:52:21.413815 25835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12c220661554431991865b1220e76e16 [sys.catalog]: This master's current role is: LEADER
I20260619 18:52:21.424755 25845 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:52:21.430577 25832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:21.410901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55141 (local address 127.3.243.126:34043)
0619 18:52:21.411092 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:21.411100 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:21.412427 (+ 1327us) server_negotiation.cc:407] Connection header received
0619 18:52:21.412755 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:21.412762 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:21.412875 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:21.412954 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:21.415358 (+ 2404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:21.416699 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:21.418792 (+ 2093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:21.419548 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:21.419655 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:21.423942 (+ 4287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:52:21.423951 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:52:21.423957 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:52:21.424016 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:52:21.427450 (+ 3434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:21.428284 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:21.428289 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:21.428291 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:21.428422 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:21.428894 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:21.428898 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:21.428900 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:21.429273 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:52:21.429540 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:21.429983 (+ 443us) server_negotiation.cc:299] Negotiation successful
0619 18:52:21.430183 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 18:52:21.431548 25845 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:52:21.441919 25845 catalog_manager.cc:1347] Generated new cluster ID: a20e83aa9f054b58af3e9463d4f1a91c
I20260619 18:52:21.442026 25845 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:52:21.463516 25845 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:52:21.465713 25845 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:52:21.479452 25845 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 12c220661554431991865b1220e76e16: Generated new TSK 0
I20260619 18:52:21.480937 25845 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:52:21.488533 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 25762
I20260619 18:52:21.490461 25824 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:52:21.588143 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 25762
W20260619 18:52:21.605777 25747 connection.cc:535] client connection to 127.3.243.126:34043 recv error: Network error: failed to read from TLS socket (remote: 127.3.243.126:34043): Connection reset by peer (error 104)
2026-06-19T18:52:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2068 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781894810715282-4045-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:52:21 dist-test-slave-m9wg krb5kdc[25861](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:52:21 dist-test-slave-m9wg krb5kdc[25861](info): set up 2 sockets
Jun 19 18:52:21 dist-test-slave-m9wg krb5kdc[25861](info): commencing operation
krb5kdc: starting...
W20260619 18:52:22.633662 4045 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.008s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781894810715282-4045-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781894810715282-4045-0/krb5kdc/test-user.keytab.
Jun 19 18:52:22 dist-test-slave-m9wg krb5kdc[25861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895142, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:52:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:52:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.243.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab.
Entry for principal kudu/127.3.243.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab.
I20260619 18:52:22.926949 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781894810715282-4045-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781894810715282-4045-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781894810715282-4045-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.126:43741
--webserver_interface=127.3.243.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab
--principal=kudu/127.3.243.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:42323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.126:43741
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:52:23.300592 25874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:52:23.300941 25874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:52:23.301083 25874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:52:23.308631 25874 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:52:23.308759 25874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:52:23.308836 25874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:52:23.308926 25874 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:52:23.308986 25874 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:52:23.319386 25874 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:42323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781894810715282-4045-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781894810715282-4045-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.243.126:43741
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab
--principal=kudu/127.3.243.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.126:43741
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.243.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781894810715282-4045-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:52:23.321108 25874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:52:23.322659 25874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:52:23.335186 25879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:52:23.335280 25882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:52:23.335211 25880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:52:23.335952 25874 server_base.cc:1034] running on GCE node
I20260619 18:52:23.336833 25874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:52:23.338353 25874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:52:23.339545 25874 hybrid_clock.cc:648] HybridClock initialized: now 1781895143339495 us; error 44 us; skew 500 ppm
I20260619 18:52:23.339884 25874 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:52:23 dist-test-slave-m9wg krb5kdc[25861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895143, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:52:23.343382 25874 init.cc:377] Logged in from keytab as kudu/127.3.243.126@KRBTEST.COM (short username kudu)
I20260619 18:52:23.345106 25874 webserver.cc:460] Webserver started at http://127.3.243.126:35185/ using document root <none> and password file <none>
I20260619 18:52:23.345891 25874 fs_manager.cc:359] Metadata directory not provided
I20260619 18:52:23.346020 25874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:52:23.346355 25874 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:52:23.350133 25874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781894810715282-4045-0/minicluster-data/master-0/data/instance:
uuid: "9ba3bc6551784561a16f3f225e9c4f17"
format_stamp: "Formatted at 2026-06-19 18:52:23 on dist-test-slave-m9wg"
server_key: "a66224252f6a765851ef94a7d33a4a82"
server_key_iv: "b52b08a1969f7a8ffcd5c1a9b4c46b62"
server_key_version: "encryptionkey@0"
I20260619 18:52:23.351075 25874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781894810715282-4045-0/minicluster-data/master-0/wal/instance:
uuid: "9ba3bc6551784561a16f3f225e9c4f17"
format_stamp: "Formatted at 2026-06-19 18:52:23 on dist-test-slave-m9wg"
server_key: "a66224252f6a765851ef94a7d33a4a82"
server_key_iv: "b52b08a1969f7a8ffcd5c1a9b4c46b62"
server_key_version: "encryptionkey@0"
I20260619 18:52:23.357842 25874 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.003s
I20260619 18:52:23.362430 25889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:52:23.364282 25874 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 18:52:23.364496 25874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781894810715282-4045-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781894810715282-4045-0/minicluster-data/master-0/wal
uuid: "9ba3bc6551784561a16f3f225e9c4f17"
format_stamp: "Formatted at 2026-06-19 18:52:23 on dist-test-slave-m9wg"
server_key: "a66224252f6a765851ef94a7d33a4a82"
server_key_iv: "b52b08a1969f7a8ffcd5c1a9b4c46b62"
server_key_version: "encryptionkey@0"
I20260619 18:52:23.364748 25874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781894810715282-4045-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781894810715282-4045-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781894810715282-4045-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:52:23.392258 25874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:52:23.396852 25874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:52:23.397148 25874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:52:23.416313 25874 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.126:43741
I20260619 18:52:23.416323 25940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.126:43741 every 8 connection(s)
I20260619 18:52:23.417857 25874 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
I20260619 18:52:23.422204 25941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:52:23.423555 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 25874
I20260619 18:52:23.423765 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781894810715282-4045-0/minicluster-data/master-0/wal/instance
I20260619 18:52:23.424494 4045 external_mini_cluster.cc:1428] Setting key 8c480e0f05405c727bc5be8df91060a8
I20260619 18:52:23.437548 25941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ba3bc6551784561a16f3f225e9c4f17: Bootstrap starting.
I20260619 18:52:23.439432 25944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:23.426543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54697 (local address 127.3.243.126:43741)
0619 18:52:23.427453 (+ 910us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:23.427467 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:23.427516 (+ 49us) server_negotiation.cc:407] Connection header received
0619 18:52:23.429382 (+ 1866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:23.429408 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:23.429893 (+ 485us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:23.430285 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:23.430864 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:23.432277 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:23.434139 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:23.434901 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:23.435028 (+ 127us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:23.436531 (+ 1503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:52:23.436540 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:52:23.436556 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:52:23.437216 (+ 660us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:52:23.437339 (+ 123us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:52:23.437375 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:52:23.437545 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:23.438143 (+ 598us) server_negotiation.cc:299] Negotiation successful
0619 18:52:23.438338 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":694,"thread_start_us":403,"threads_started":1}
I20260619 18:52:23.441507 25941 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9ba3bc6551784561a16f3f225e9c4f17: Neither blocks nor log segments found. Creating new log.
I20260619 18:52:23.442806 25941 log.cc:824] T 00000000000000000000000000000000 P 9ba3bc6551784561a16f3f225e9c4f17: Log is configured to *not* fsync() on all Append() calls
I20260619 18:52:23.443547 4045 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:52:23.445684 25941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ba3bc6551784561a16f3f225e9c4f17: No bootstrap required, opened a new log
I20260619 18:52:23.454185 25941 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ba3bc6551784561a16f3f225e9c4f17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ba3bc6551784561a16f3f225e9c4f17" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 43741 } }
I20260619 18:52:23.454614 25941 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9ba3bc6551784561a16f3f225e9c4f17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:52:23.454789 25941 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ba3bc6551784561a16f3f225e9c4f17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ba3bc6551784561a16f3f225e9c4f17, State: Initialized, Role: FOLLOWER
I20260619 18:52:23.455062 25944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:23.448126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48735 (local address 127.3.243.126:43741)
0619 18:52:23.448297 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:23.448301 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:23.448810 (+ 509us) server_negotiation.cc:407] Connection header received
0619 18:52:23.448990 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:23.448994 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:23.449058 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:23.449149 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:23.449676 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:23.450641 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:23.452180 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:23.452726 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:23.452867 (+ 141us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:23.454200 (+ 1333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:52:23.454206 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:52:23.454210 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:52:23.454229 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:52:23.454282 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:52:23.454299 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:52:23.454410 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:23.454648 (+ 238us) server_negotiation.cc:299] Negotiation successful
0619 18:52:23.454708 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 18:52:23.455513 25941 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ba3bc6551784561a16f3f225e9c4f17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ba3bc6551784561a16f3f225e9c4f17" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 43741 } }
I20260619 18:52:23.455772 25941 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9ba3bc6551784561a16f3f225e9c4f17 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:52:23.455907 25941 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9ba3bc6551784561a16f3f225e9c4f17 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:52:23.456074 25941 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ba3bc6551784561a16f3f225e9c4f17 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:52:23.458774 25941 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9ba3bc6551784561a16f3f225e9c4f17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ba3bc6551784561a16f3f225e9c4f17" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 43741 } }
I20260619 18:52:23.459357 25941 leader_election.cc:302] T 00000000000000000000000000000000 P 9ba3bc6551784561a16f3f225e9c4f17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9ba3bc6551784561a16f3f225e9c4f17; no voters:
I20260619 18:52:23.459928 25941 leader_election.cc:288] T 00000000000000000000000000000000 P 9ba3bc6551784561a16f3f225e9c4f17 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:52:23.460052 25951 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9ba3bc6551784561a16f3f225e9c4f17 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:52:23.460783 25951 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9ba3bc6551784561a16f3f225e9c4f17 [term 1 LEADER]: Becoming Leader. State: Replica: 9ba3bc6551784561a16f3f225e9c4f17, State: Running, Role: LEADER
I20260619 18:52:23.461485 25951 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ba3bc6551784561a16f3f225e9c4f17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ba3bc6551784561a16f3f225e9c4f17" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 43741 } }
I20260619 18:52:23.462416 25941 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ba3bc6551784561a16f3f225e9c4f17 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:52:23.464702 25952 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ba3bc6551784561a16f3f225e9c4f17 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9ba3bc6551784561a16f3f225e9c4f17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ba3bc6551784561a16f3f225e9c4f17" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 43741 } } }
I20260619 18:52:23.465047 25952 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ba3bc6551784561a16f3f225e9c4f17 [sys.catalog]: This master's current role is: LEADER
I20260619 18:52:23.465533 25953 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ba3bc6551784561a16f3f225e9c4f17 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ba3bc6551784561a16f3f225e9c4f17. Latest consensus state: current_term: 1 leader_uuid: "9ba3bc6551784561a16f3f225e9c4f17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ba3bc6551784561a16f3f225e9c4f17" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 43741 } } }
I20260619 18:52:23.465880 25953 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ba3bc6551784561a16f3f225e9c4f17 [sys.catalog]: This master's current role is: LEADER
I20260619 18:52:23.466038 25957 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:52:23.472752 25957 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:52:23.485441 25957 catalog_manager.cc:1347] Generated new cluster ID: 47d4bbe4f26147cea5bc98b521db996b
I20260619 18:52:23.485618 25957 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:52:23.511407 25957 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:52:23.513301 25957 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:52:23.524534 25957 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9ba3bc6551784561a16f3f225e9c4f17: Generated new TSK 0
I20260619 18:52:23.525645 25957 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:52:23.536931 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 25874
W20260619 18:52:23.538878 25936 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:54697
W20260619 18:52:23.539659 4045 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 25874: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 18:52:23.539849 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 25874
W20260619 18:52:23.557575 25859 connection.cc:535] client connection to 127.3.243.126:43741 recv error: Network error: failed to read from TLS socket (remote: 127.3.243.126:43741): Connection reset by peer (error 104)
2026-06-19T18:52:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1952 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781894810715282-4045-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:52:23 dist-test-slave-m9wg krb5kdc[25972](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:52:23 dist-test-slave-m9wg krb5kdc[25972](info): set up 2 sockets
Jun 19 18:52:23 dist-test-slave-m9wg krb5kdc[25972](info): commencing operation
krb5kdc: starting...
W20260619 18:52:24.592942 4045 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.042s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781894810715282-4045-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781894810715282-4045-0/krb5kdc/test-user.keytab.
Jun 19 18:52:24 dist-test-slave-m9wg krb5kdc[25972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895144, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:52:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:52:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.243.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab.
Entry for principal kudu/127.3.243.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab.
I20260619 18:52:24.865111 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781894810715282-4045-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781894810715282-4045-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781894810715282-4045-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.126:39165
--webserver_interface=127.3.243.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab
--principal=kudu/127.3.243.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:35279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.126:39165
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:52:25.243619 25985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:52:25.243973 25985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:52:25.244081 25985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:52:25.251619 25985 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:52:25.251780 25985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:52:25.251838 25985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:52:25.251902 25985 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:52:25.251950 25985 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:52:25.262956 25985 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:35279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781894810715282-4045-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781894810715282-4045-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.243.126:39165
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab
--principal=kudu/127.3.243.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.126:39165
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.243.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781894810715282-4045-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:52:25.264775 25985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:52:25.266431 25985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:52:25.280421 25991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:52:25.280480 25990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:52:25.280598 25993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:52:25.281351 25985 server_base.cc:1034] running on GCE node
I20260619 18:52:25.282315 25985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:52:25.284247 25985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:52:25.285475 25985 hybrid_clock.cc:648] HybridClock initialized: now 1781895145285410 us; error 64 us; skew 500 ppm
I20260619 18:52:25.285866 25985 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:52:25 dist-test-slave-m9wg krb5kdc[25972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895145, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:52:25.298257 25985 init.cc:377] Logged in from keytab as kudu/127.3.243.126@KRBTEST.COM (short username kudu)
I20260619 18:52:25.300344 25985 webserver.cc:460] Webserver started at http://127.3.243.126:46339/ using document root <none> and password file <none>
I20260619 18:52:25.301433 25985 fs_manager.cc:359] Metadata directory not provided
I20260619 18:52:25.301550 25985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:52:25.301862 25985 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:52:25.305536 25985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781894810715282-4045-0/minicluster-data/master-0/data/instance:
uuid: "221c70c636874474b0984af9a8c280eb"
format_stamp: "Formatted at 2026-06-19 18:52:25 on dist-test-slave-m9wg"
server_key: "4430ac3d5fa37c62820ac1164ef4f1c7"
server_key_iv: "206c064ca9e21cb20a9a28bf124a1193"
server_key_version: "encryptionkey@0"
I20260619 18:52:25.306327 25985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781894810715282-4045-0/minicluster-data/master-0/wal/instance:
uuid: "221c70c636874474b0984af9a8c280eb"
format_stamp: "Formatted at 2026-06-19 18:52:25 on dist-test-slave-m9wg"
server_key: "4430ac3d5fa37c62820ac1164ef4f1c7"
server_key_iv: "206c064ca9e21cb20a9a28bf124a1193"
server_key_version: "encryptionkey@0"
I20260619 18:52:25.312281 25985 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260619 18:52:25.316267 26000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:52:25.317772 25985 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 18:52:25.317945 25985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781894810715282-4045-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781894810715282-4045-0/minicluster-data/master-0/wal
uuid: "221c70c636874474b0984af9a8c280eb"
format_stamp: "Formatted at 2026-06-19 18:52:25 on dist-test-slave-m9wg"
server_key: "4430ac3d5fa37c62820ac1164ef4f1c7"
server_key_iv: "206c064ca9e21cb20a9a28bf124a1193"
server_key_version: "encryptionkey@0"
I20260619 18:52:25.318158 25985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781894810715282-4045-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781894810715282-4045-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781894810715282-4045-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:52:25.324340 25985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:52:25.328446 25985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:52:25.328719 25985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:52:25.348354 25985 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.126:39165
I20260619 18:52:25.348361 26051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.126:39165 every 8 connection(s)
I20260619 18:52:25.350096 25985 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
I20260619 18:52:25.355228 26052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:52:25.356027 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 25985
I20260619 18:52:25.356277 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781894810715282-4045-0/minicluster-data/master-0/wal/instance
I20260619 18:52:25.356814 4045 external_mini_cluster.cc:1428] Setting key 6e1a861775895648a820eb3c64dedbed
I20260619 18:52:25.367856 26055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:25.359166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45509 (local address 127.3.243.126:39165)
0619 18:52:25.360276 (+ 1110us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:25.360295 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:25.360352 (+ 57us) server_negotiation.cc:407] Connection header received
0619 18:52:25.362855 (+ 2503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:25.362905 (+ 50us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:25.363588 (+ 683us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:25.364192 (+ 604us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:25.365662 (+ 1470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:52:25.365671 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:52:25.365693 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:52:25.365715 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:52:25.365833 (+ 118us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:52:25.365859 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:52:25.366088 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:25.366242 (+ 154us) server_negotiation.cc:299] Negotiation successful
0619 18:52:25.366513 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":771,"thread_start_us":446,"threads_started":1}
I20260619 18:52:25.372680 26052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 221c70c636874474b0984af9a8c280eb: Bootstrap starting.
I20260619 18:52:25.373740 4045 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:52:25.376971 26052 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 221c70c636874474b0984af9a8c280eb: Neither blocks nor log segments found. Creating new log.
I20260619 18:52:25.378389 26052 log.cc:824] T 00000000000000000000000000000000 P 221c70c636874474b0984af9a8c280eb: Log is configured to *not* fsync() on all Append() calls
I20260619 18:52:25.381459 26055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:25.378712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51443 (local address 127.3.243.126:39165)
0619 18:52:25.378935 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:25.378944 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:25.379379 (+ 435us) server_negotiation.cc:407] Connection header received
0619 18:52:25.379582 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:25.379587 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:25.379651 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:25.379759 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:25.380774 (+ 1015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:52:25.380779 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:52:25.380782 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:52:25.380799 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:52:25.380844 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:52:25.380864 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:52:25.380968 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:25.381096 (+ 128us) server_negotiation.cc:299] Negotiation successful
0619 18:52:25.381169 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 18:52:25.382154 26052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 221c70c636874474b0984af9a8c280eb: No bootstrap required, opened a new log
I20260619 18:52:25.391002 26052 raft_consensus.cc:348] T 00000000000000000000000000000000 P 221c70c636874474b0984af9a8c280eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "221c70c636874474b0984af9a8c280eb" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 39165 } }
I20260619 18:52:25.391511 26052 raft_consensus.cc:374] T 00000000000000000000000000000000 P 221c70c636874474b0984af9a8c280eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:52:25.391601 26052 raft_consensus.cc:729] T 00000000000000000000000000000000 P 221c70c636874474b0984af9a8c280eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 221c70c636874474b0984af9a8c280eb, State: Initialized, Role: FOLLOWER
I20260619 18:52:25.392352 26052 consensus_queue.cc:260] T 00000000000000000000000000000000 P 221c70c636874474b0984af9a8c280eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "221c70c636874474b0984af9a8c280eb" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 39165 } }
I20260619 18:52:25.392606 26052 raft_consensus.cc:388] T 00000000000000000000000000000000 P 221c70c636874474b0984af9a8c280eb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:52:25.392742 26052 raft_consensus.cc:482] T 00000000000000000000000000000000 P 221c70c636874474b0984af9a8c280eb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:52:25.392890 26052 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 221c70c636874474b0984af9a8c280eb [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:52:25.395884 26052 raft_consensus.cc:504] T 00000000000000000000000000000000 P 221c70c636874474b0984af9a8c280eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "221c70c636874474b0984af9a8c280eb" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 39165 } }
I20260619 18:52:25.396986 26052 leader_election.cc:302] T 00000000000000000000000000000000 P 221c70c636874474b0984af9a8c280eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 221c70c636874474b0984af9a8c280eb; no voters:
I20260619 18:52:25.397781 26052 leader_election.cc:288] T 00000000000000000000000000000000 P 221c70c636874474b0984af9a8c280eb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:52:25.397930 26062 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 221c70c636874474b0984af9a8c280eb [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:52:25.399013 26062 raft_consensus.cc:686] T 00000000000000000000000000000000 P 221c70c636874474b0984af9a8c280eb [term 1 LEADER]: Becoming Leader. State: Replica: 221c70c636874474b0984af9a8c280eb, State: Running, Role: LEADER
I20260619 18:52:25.399603 26062 consensus_queue.cc:237] T 00000000000000000000000000000000 P 221c70c636874474b0984af9a8c280eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "221c70c636874474b0984af9a8c280eb" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 39165 } }
I20260619 18:52:25.400345 26052 sys_catalog.cc:564] T 00000000000000000000000000000000 P 221c70c636874474b0984af9a8c280eb [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:52:25.402966 26064 sys_catalog.cc:455] T 00000000000000000000000000000000 P 221c70c636874474b0984af9a8c280eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 221c70c636874474b0984af9a8c280eb. Latest consensus state: current_term: 1 leader_uuid: "221c70c636874474b0984af9a8c280eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "221c70c636874474b0984af9a8c280eb" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 39165 } } }
I20260619 18:52:25.403640 26064 sys_catalog.cc:458] T 00000000000000000000000000000000 P 221c70c636874474b0984af9a8c280eb [sys.catalog]: This master's current role is: LEADER
I20260619 18:52:25.403005 26063 sys_catalog.cc:455] T 00000000000000000000000000000000 P 221c70c636874474b0984af9a8c280eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "221c70c636874474b0984af9a8c280eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "221c70c636874474b0984af9a8c280eb" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 39165 } } }
I20260619 18:52:25.404276 26063 sys_catalog.cc:458] T 00000000000000000000000000000000 P 221c70c636874474b0984af9a8c280eb [sys.catalog]: This master's current role is: LEADER
I20260619 18:52:25.405997 26069 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:52:25.411924 26069 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:52:25.422564 26069 catalog_manager.cc:1347] Generated new cluster ID: 8e55b33c0133486db29c5afa754a68ef
I20260619 18:52:25.422710 26069 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:52:25.457966 26069 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:52:25.458570 26069 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:52:25.467932 26069 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 221c70c636874474b0984af9a8c280eb: Generated new TSK 0
I20260619 18:52:25.468958 26069 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:52:25.474615 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 25985
W20260619 18:52:25.475723 26047 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:45509
W20260619 18:52:25.476471 4045 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 25985: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 18:52:25.476611 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 25985
2026-06-19T18:52:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1935 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781894810715282-4045-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:52:25 dist-test-slave-m9wg krb5kdc[26083](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:52:25 dist-test-slave-m9wg krb5kdc[26083](info): set up 2 sockets
Jun 19 18:52:25 dist-test-slave-m9wg krb5kdc[26083](info): commencing operation
krb5kdc: starting...
W20260619 18:52:26.504812 4045 mini_kdc.cc:121] Time spent starting KDC: real 0.991s user 0.004s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781894810715282-4045-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781894810715282-4045-0/krb5kdc/test-user.keytab.
Jun 19 18:52:26 dist-test-slave-m9wg krb5kdc[26083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895146, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:52:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:52:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.243.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab.
Entry for principal kudu/127.3.243.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab.
I20260619 18:52:26.767094 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781894810715282-4045-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781894810715282-4045-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781894810715282-4045-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.126:45659
--webserver_interface=127.3.243.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab
--principal=kudu/127.3.243.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:42539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.126:45659
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:52:27.149900 26096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:52:27.150287 26096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:52:27.150449 26096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:52:27.158243 26096 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:52:27.158408 26096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:52:27.158500 26096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:52:27.158599 26096 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:52:27.158672 26096 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:52:27.169557 26096 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:42539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781894810715282-4045-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781894810715282-4045-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.243.126:45659
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab
--principal=kudu/127.3.243.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.126:45659
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.243.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781894810715282-4045-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:52:27.171298 26096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:52:27.172968 26096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:52:27.186012 26102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:52:27.185921 26101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:52:27.185986 26104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:52:27.187019 26096 server_base.cc:1034] running on GCE node
I20260619 18:52:27.187819 26096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:52:27.189385 26096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:52:27.190588 26096 hybrid_clock.cc:648] HybridClock initialized: now 1781895147190531 us; error 51 us; skew 500 ppm
I20260619 18:52:27.190940 26096 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:52:27 dist-test-slave-m9wg krb5kdc[26083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895147, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:52:27.194404 26096 init.cc:377] Logged in from keytab as kudu/127.3.243.126@KRBTEST.COM (short username kudu)
I20260619 18:52:27.196007 26096 webserver.cc:460] Webserver started at http://127.3.243.126:40395/ using document root <none> and password file <none>
I20260619 18:52:27.196797 26096 fs_manager.cc:359] Metadata directory not provided
I20260619 18:52:27.196919 26096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:52:27.197248 26096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:52:27.201625 26096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781894810715282-4045-0/minicluster-data/master-0/data/instance:
uuid: "fecccc5a86984e0fb2e213edafb35299"
format_stamp: "Formatted at 2026-06-19 18:52:27 on dist-test-slave-m9wg"
server_key: "958702d1a895b1b6d0b13488e376bfbe"
server_key_iv: "7be2a526bcea1a2720a3186bd7432ae6"
server_key_version: "encryptionkey@0"
I20260619 18:52:27.202680 26096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781894810715282-4045-0/minicluster-data/master-0/wal/instance:
uuid: "fecccc5a86984e0fb2e213edafb35299"
format_stamp: "Formatted at 2026-06-19 18:52:27 on dist-test-slave-m9wg"
server_key: "958702d1a895b1b6d0b13488e376bfbe"
server_key_iv: "7be2a526bcea1a2720a3186bd7432ae6"
server_key_version: "encryptionkey@0"
I20260619 18:52:27.209867 26096 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260619 18:52:27.213986 26111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:52:27.215665 26096 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 18:52:27.215878 26096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781894810715282-4045-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781894810715282-4045-0/minicluster-data/master-0/wal
uuid: "fecccc5a86984e0fb2e213edafb35299"
format_stamp: "Formatted at 2026-06-19 18:52:27 on dist-test-slave-m9wg"
server_key: "958702d1a895b1b6d0b13488e376bfbe"
server_key_iv: "7be2a526bcea1a2720a3186bd7432ae6"
server_key_version: "encryptionkey@0"
I20260619 18:52:27.216115 26096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781894810715282-4045-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781894810715282-4045-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781894810715282-4045-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:52:27.246484 26096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:52:27.252060 26096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:52:27.252424 26096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:52:27.271684 26096 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.126:45659
I20260619 18:52:27.271692 26162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.126:45659 every 8 connection(s)
I20260619 18:52:27.273428 26096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
I20260619 18:52:27.278352 26163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:52:27.279922 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 26096
I20260619 18:52:27.280128 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781894810715282-4045-0/minicluster-data/master-0/wal/instance
I20260619 18:52:27.280824 4045 external_mini_cluster.cc:1428] Setting key bfad28fb82bf9b9cfa9b1ea2c95c9594
I20260619 18:52:27.294216 26163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fecccc5a86984e0fb2e213edafb35299: Bootstrap starting.
Jun 19 18:52:27 dist-test-slave-m9wg krb5kdc[26083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895146, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.126@KRBTEST.COM
I20260619 18:52:27.298640 26163 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fecccc5a86984e0fb2e213edafb35299: Neither blocks nor log segments found. Creating new log.
I20260619 18:52:27.300187 26163 log.cc:824] T 00000000000000000000000000000000 P fecccc5a86984e0fb2e213edafb35299: Log is configured to *not* fsync() on all Append() calls
I20260619 18:52:27.303766 26163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fecccc5a86984e0fb2e213edafb35299: No bootstrap required, opened a new log
I20260619 18:52:27.306258 26166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:27.283022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60153 (local address 127.3.243.126:45659)
0619 18:52:27.283866 (+ 844us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:27.283883 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:27.283933 (+ 50us) server_negotiation.cc:407] Connection header received
0619 18:52:27.285737 (+ 1804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:27.285763 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:27.286274 (+ 511us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:27.286663 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:27.288455 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:27.289856 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:27.291785 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:27.292553 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:27.292686 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:27.297871 (+ 5185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:52:27.297879 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:52:27.297896 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:52:27.297942 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:52:27.301606 (+ 3664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:27.302284 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:27.302294 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:27.302302 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:27.302512 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:27.302860 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:27.302864 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:27.302866 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:27.303568 (+ 702us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:52:27.303882 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:27.304241 (+ 359us) server_negotiation.cc:299] Negotiation successful
0619 18:52:27.304777 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":383,"threads_started":1}
I20260619 18:52:27.310577 4045 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:52:27.313304 26163 raft_consensus.cc:348] T 00000000000000000000000000000000 P fecccc5a86984e0fb2e213edafb35299 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fecccc5a86984e0fb2e213edafb35299" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 45659 } }
I20260619 18:52:27.313783 26163 raft_consensus.cc:374] T 00000000000000000000000000000000 P fecccc5a86984e0fb2e213edafb35299 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:52:27.313925 26163 raft_consensus.cc:729] T 00000000000000000000000000000000 P fecccc5a86984e0fb2e213edafb35299 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fecccc5a86984e0fb2e213edafb35299, State: Initialized, Role: FOLLOWER
I20260619 18:52:27.314594 26163 consensus_queue.cc:260] T 00000000000000000000000000000000 P fecccc5a86984e0fb2e213edafb35299 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fecccc5a86984e0fb2e213edafb35299" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 45659 } }
I20260619 18:52:27.314829 26163 raft_consensus.cc:388] T 00000000000000000000000000000000 P fecccc5a86984e0fb2e213edafb35299 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:52:27.314967 26163 raft_consensus.cc:482] T 00000000000000000000000000000000 P fecccc5a86984e0fb2e213edafb35299 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:52:27.315176 26163 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fecccc5a86984e0fb2e213edafb35299 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:52:27.317788 26163 raft_consensus.cc:504] T 00000000000000000000000000000000 P fecccc5a86984e0fb2e213edafb35299 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fecccc5a86984e0fb2e213edafb35299" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 45659 } }
I20260619 18:52:27.318323 26163 leader_election.cc:302] T 00000000000000000000000000000000 P fecccc5a86984e0fb2e213edafb35299 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fecccc5a86984e0fb2e213edafb35299; no voters:
I20260619 18:52:27.318938 26163 leader_election.cc:288] T 00000000000000000000000000000000 P fecccc5a86984e0fb2e213edafb35299 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:52:27.319048 26173 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fecccc5a86984e0fb2e213edafb35299 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:52:27.320416 26173 raft_consensus.cc:686] T 00000000000000000000000000000000 P fecccc5a86984e0fb2e213edafb35299 [term 1 LEADER]: Becoming Leader. State: Replica: fecccc5a86984e0fb2e213edafb35299, State: Running, Role: LEADER
I20260619 18:52:27.321096 26173 consensus_queue.cc:237] T 00000000000000000000000000000000 P fecccc5a86984e0fb2e213edafb35299 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fecccc5a86984e0fb2e213edafb35299" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 45659 } }
I20260619 18:52:27.321578 26163 sys_catalog.cc:564] T 00000000000000000000000000000000 P fecccc5a86984e0fb2e213edafb35299 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:52:27.324519 26175 sys_catalog.cc:455] T 00000000000000000000000000000000 P fecccc5a86984e0fb2e213edafb35299 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fecccc5a86984e0fb2e213edafb35299. Latest consensus state: current_term: 1 leader_uuid: "fecccc5a86984e0fb2e213edafb35299" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fecccc5a86984e0fb2e213edafb35299" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 45659 } } }
I20260619 18:52:27.324949 26175 sys_catalog.cc:458] T 00000000000000000000000000000000 P fecccc5a86984e0fb2e213edafb35299 [sys.catalog]: This master's current role is: LEADER
I20260619 18:52:27.325605 26174 sys_catalog.cc:455] T 00000000000000000000000000000000 P fecccc5a86984e0fb2e213edafb35299 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fecccc5a86984e0fb2e213edafb35299" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fecccc5a86984e0fb2e213edafb35299" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 45659 } } }
I20260619 18:52:27.325894 26174 sys_catalog.cc:458] T 00000000000000000000000000000000 P fecccc5a86984e0fb2e213edafb35299 [sys.catalog]: This master's current role is: LEADER
I20260619 18:52:27.328838 26181 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:52:27.331947 26166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:27.315376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46643 (local address 127.3.243.126:45659)
0619 18:52:27.315560 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:27.315567 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:27.315987 (+ 420us) server_negotiation.cc:407] Connection header received
0619 18:52:27.316193 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:27.316199 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:27.316288 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:27.316380 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:27.317852 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:27.318709 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:27.320377 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:27.321035 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:27.321120 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:27.324315 (+ 3195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:52:27.324322 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:52:27.324327 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:52:27.324369 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:52:27.328308 (+ 3939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:27.329076 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:27.329085 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:27.329088 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:27.329235 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:27.329745 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:27.329749 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:27.329752 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:27.330224 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:52:27.331166 (+ 942us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:27.331330 (+ 164us) server_negotiation.cc:299] Negotiation successful
0619 18:52:27.331570 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 18:52:27.335012 26181 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:52:27.344286 26181 catalog_manager.cc:1347] Generated new cluster ID: c81ac91a17244d7999e254b38246a253
I20260619 18:52:27.344401 26181 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:52:27.365593 26181 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:52:27.367700 26181 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:52:27.378950 26181 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fecccc5a86984e0fb2e213edafb35299: Generated new TSK 0
I20260619 18:52:27.380357 26181 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:52:27.390407 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 26096
I20260619 18:52:27.392558 26158 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:52:27.491739 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 26096
2026-06-19T18:52:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2014 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781894810715282-4045-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:52:27 dist-test-slave-m9wg krb5kdc[26195](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:52:27 dist-test-slave-m9wg krb5kdc[26195](info): set up 2 sockets
Jun 19 18:52:27 dist-test-slave-m9wg krb5kdc[26195](info): commencing operation
krb5kdc: starting...
W20260619 18:52:28.525003 4045 mini_kdc.cc:121] Time spent starting KDC: real 0.997s user 0.000s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781894810715282-4045-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781894810715282-4045-0/krb5kdc/test-user.keytab.
Jun 19 18:52:28 dist-test-slave-m9wg krb5kdc[26195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895148, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:52:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:52:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.243.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab.
Entry for principal kudu/127.3.243.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab.
I20260619 18:52:28.821507 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781894810715282-4045-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781894810715282-4045-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781894810715282-4045-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.126:44513
--webserver_interface=127.3.243.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab
--principal=kudu/127.3.243.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:35371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.126:44513
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:52:29.198185 26208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:52:29.198587 26208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:52:29.198702 26208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:52:29.206611 26208 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:52:29.206760 26208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:52:29.206810 26208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:52:29.206876 26208 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:52:29.206928 26208 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:52:29.218663 26208 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:35371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781894810715282-4045-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781894810715282-4045-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.243.126:44513
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab
--principal=kudu/127.3.243.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.126:44513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.243.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781894810715282-4045-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:52:29.220275 26208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:52:29.221863 26208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:52:29.234416 26216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:52:29.234584 26213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:52:29.234542 26214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:52:29.235481 26208 server_base.cc:1034] running on GCE node
I20260619 18:52:29.236358 26208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:52:29.237944 26208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:52:29.239144 26208 hybrid_clock.cc:648] HybridClock initialized: now 1781895149239088 us; error 51 us; skew 500 ppm
I20260619 18:52:29.239504 26208 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:52:29 dist-test-slave-m9wg krb5kdc[26195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895149, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:52:29.243116 26208 init.cc:377] Logged in from keytab as kudu/127.3.243.126@KRBTEST.COM (short username kudu)
I20260619 18:52:29.244728 26208 webserver.cc:460] Webserver started at http://127.3.243.126:40459/ using document root <none> and password file <none>
I20260619 18:52:29.245500 26208 fs_manager.cc:359] Metadata directory not provided
I20260619 18:52:29.245618 26208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:52:29.245924 26208 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:52:29.249287 26208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781894810715282-4045-0/minicluster-data/master-0/data/instance:
uuid: "bdba9a0e65a64778a2cf585dfeda1bca"
format_stamp: "Formatted at 2026-06-19 18:52:29 on dist-test-slave-m9wg"
server_key: "03bf13ffd1a4922f52809dbeca356e6c"
server_key_iv: "bf9eba79a06438a265917e3af400ed5c"
server_key_version: "encryptionkey@0"
I20260619 18:52:29.250106 26208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781894810715282-4045-0/minicluster-data/master-0/wal/instance:
uuid: "bdba9a0e65a64778a2cf585dfeda1bca"
format_stamp: "Formatted at 2026-06-19 18:52:29 on dist-test-slave-m9wg"
server_key: "03bf13ffd1a4922f52809dbeca356e6c"
server_key_iv: "bf9eba79a06438a265917e3af400ed5c"
server_key_version: "encryptionkey@0"
I20260619 18:52:29.255746 26208 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 18:52:29.259461 26223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:52:29.260936 26208 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260619 18:52:29.261096 26208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781894810715282-4045-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781894810715282-4045-0/minicluster-data/master-0/wal
uuid: "bdba9a0e65a64778a2cf585dfeda1bca"
format_stamp: "Formatted at 2026-06-19 18:52:29 on dist-test-slave-m9wg"
server_key: "03bf13ffd1a4922f52809dbeca356e6c"
server_key_iv: "bf9eba79a06438a265917e3af400ed5c"
server_key_version: "encryptionkey@0"
I20260619 18:52:29.261289 26208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781894810715282-4045-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781894810715282-4045-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781894810715282-4045-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:52:29.285781 26208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:52:29.290033 26208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:52:29.290323 26208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:52:29.309769 26208 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.126:44513
I20260619 18:52:29.309789 26274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.126:44513 every 8 connection(s)
I20260619 18:52:29.311443 26208 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
I20260619 18:52:29.312062 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 26208
I20260619 18:52:29.312297 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781894810715282-4045-0/minicluster-data/master-0/wal/instance
I20260619 18:52:29.312920 4045 external_mini_cluster.cc:1428] Setting key 299539d5fb8eb80578aab794e01f4446
I20260619 18:52:29.317169 26275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:52:29.330610 26278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:29.315166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37267 (local address 127.3.243.126:44513)
0619 18:52:29.316103 (+ 937us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:29.316122 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:29.316221 (+ 99us) server_negotiation.cc:407] Connection header received
0619 18:52:29.318685 (+ 2464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:29.318724 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:29.319339 (+ 615us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:29.319849 (+ 510us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:29.320477 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:29.322580 (+ 2103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:29.324036 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:29.325271 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:29.325459 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:29.326887 (+ 1428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:52:29.326898 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:52:29.326924 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:52:29.327822 (+ 898us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:52:29.327967 (+ 145us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:52:29.328008 (+ 41us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:52:29.328142 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:29.328444 (+ 302us) server_negotiation.cc:299] Negotiation successful
0619 18:52:29.328839 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":657,"thread_start_us":381,"threads_started":1}
I20260619 18:52:29.333169 26275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bdba9a0e65a64778a2cf585dfeda1bca: Bootstrap starting.
I20260619 18:52:29.335438 4045 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:52:29.337594 26275 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bdba9a0e65a64778a2cf585dfeda1bca: Neither blocks nor log segments found. Creating new log.
I20260619 18:52:29.339056 26275 log.cc:824] T 00000000000000000000000000000000 P bdba9a0e65a64778a2cf585dfeda1bca: Log is configured to *not* fsync() on all Append() calls
I20260619 18:52:29.342856 26275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bdba9a0e65a64778a2cf585dfeda1bca: No bootstrap required, opened a new log
I20260619 18:52:29.347582 26278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:29.340997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39231 (local address 127.3.243.126:44513)
0619 18:52:29.341178 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:29.341191 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:29.341632 (+ 441us) server_negotiation.cc:407] Connection header received
0619 18:52:29.341818 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:29.341823 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:29.341905 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:29.342016 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:29.342506 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:29.343634 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:29.344993 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:29.345697 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:29.345826 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:29.346849 (+ 1023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:52:29.346855 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:52:29.346861 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:52:29.346886 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:52:29.346940 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:52:29.346960 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:52:29.347054 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:29.347130 (+ 76us) server_negotiation.cc:299] Negotiation successful
0619 18:52:29.347231 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 18:52:29.351931 26275 raft_consensus.cc:348] T 00000000000000000000000000000000 P bdba9a0e65a64778a2cf585dfeda1bca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdba9a0e65a64778a2cf585dfeda1bca" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 44513 } }
I20260619 18:52:29.352361 26275 raft_consensus.cc:374] T 00000000000000000000000000000000 P bdba9a0e65a64778a2cf585dfeda1bca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:52:29.352469 26275 raft_consensus.cc:729] T 00000000000000000000000000000000 P bdba9a0e65a64778a2cf585dfeda1bca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdba9a0e65a64778a2cf585dfeda1bca, State: Initialized, Role: FOLLOWER
I20260619 18:52:29.353152 26275 consensus_queue.cc:260] T 00000000000000000000000000000000 P bdba9a0e65a64778a2cf585dfeda1bca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdba9a0e65a64778a2cf585dfeda1bca" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 44513 } }
I20260619 18:52:29.353436 26275 raft_consensus.cc:388] T 00000000000000000000000000000000 P bdba9a0e65a64778a2cf585dfeda1bca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:52:29.353555 26275 raft_consensus.cc:482] T 00000000000000000000000000000000 P bdba9a0e65a64778a2cf585dfeda1bca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:52:29.353700 26275 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bdba9a0e65a64778a2cf585dfeda1bca [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:52:29.356073 26275 raft_consensus.cc:504] T 00000000000000000000000000000000 P bdba9a0e65a64778a2cf585dfeda1bca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdba9a0e65a64778a2cf585dfeda1bca" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 44513 } }
I20260619 18:52:29.356724 26275 leader_election.cc:302] T 00000000000000000000000000000000 P bdba9a0e65a64778a2cf585dfeda1bca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bdba9a0e65a64778a2cf585dfeda1bca; no voters:
I20260619 18:52:29.357311 26275 leader_election.cc:288] T 00000000000000000000000000000000 P bdba9a0e65a64778a2cf585dfeda1bca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:52:29.357422 26285 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bdba9a0e65a64778a2cf585dfeda1bca [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:52:29.358085 26285 raft_consensus.cc:686] T 00000000000000000000000000000000 P bdba9a0e65a64778a2cf585dfeda1bca [term 1 LEADER]: Becoming Leader. State: Replica: bdba9a0e65a64778a2cf585dfeda1bca, State: Running, Role: LEADER
I20260619 18:52:29.358645 26285 consensus_queue.cc:237] T 00000000000000000000000000000000 P bdba9a0e65a64778a2cf585dfeda1bca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdba9a0e65a64778a2cf585dfeda1bca" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 44513 } }
I20260619 18:52:29.359452 26275 sys_catalog.cc:564] T 00000000000000000000000000000000 P bdba9a0e65a64778a2cf585dfeda1bca [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:52:29.361315 26287 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdba9a0e65a64778a2cf585dfeda1bca [sys.catalog]: SysCatalogTable state changed. Reason: New leader bdba9a0e65a64778a2cf585dfeda1bca. Latest consensus state: current_term: 1 leader_uuid: "bdba9a0e65a64778a2cf585dfeda1bca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdba9a0e65a64778a2cf585dfeda1bca" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 44513 } } }
I20260619 18:52:29.361752 26287 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdba9a0e65a64778a2cf585dfeda1bca [sys.catalog]: This master's current role is: LEADER
I20260619 18:52:29.362558 26290 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:52:29.362416 26286 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdba9a0e65a64778a2cf585dfeda1bca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bdba9a0e65a64778a2cf585dfeda1bca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdba9a0e65a64778a2cf585dfeda1bca" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 44513 } } }
I20260619 18:52:29.363605 26286 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdba9a0e65a64778a2cf585dfeda1bca [sys.catalog]: This master's current role is: LEADER
I20260619 18:52:29.369158 26290 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:52:29.379503 26290 catalog_manager.cc:1347] Generated new cluster ID: 758d8f32f0c74664830a54ee55a2db98
I20260619 18:52:29.379604 26290 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:52:29.390556 26290 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:52:29.391875 26290 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:52:29.405529 26290 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bdba9a0e65a64778a2cf585dfeda1bca: Generated new TSK 0
I20260619 18:52:29.406598 26290 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:52:29.421834 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 26208
W20260619 18:52:29.423117 26270 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:37267
W20260619 18:52:29.423919 4045 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 26208: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 18:52:29.424062 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 26208
2026-06-19T18:52:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1931 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781894810715282-4045-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:52:29 dist-test-slave-m9wg krb5kdc[26306](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:52:29 dist-test-slave-m9wg krb5kdc[26306](info): set up 2 sockets
Jun 19 18:52:29 dist-test-slave-m9wg krb5kdc[26306](info): commencing operation
krb5kdc: starting...
W20260619 18:52:30.472995 4045 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.001s sys 0.035s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781894810715282-4045-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781894810715282-4045-0/krb5kdc/test-user.keytab.
Jun 19 18:52:30 dist-test-slave-m9wg krb5kdc[26306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895150, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:52:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:52:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.243.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab.
Entry for principal kudu/127.3.243.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab.
I20260619 18:52:30.787106 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781894810715282-4045-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781894810715282-4045-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781894810715282-4045-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.126:38397
--webserver_interface=127.3.243.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab
--principal=kudu/127.3.243.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:35895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.126:38397
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:52:31.174949 26319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:52:31.175319 26319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:52:31.175432 26319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:52:31.183041 26319 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:52:31.183172 26319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:52:31.183236 26319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:52:31.183301 26319 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:52:31.183347 26319 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:52:31.193948 26319 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:35895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781894810715282-4045-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781894810715282-4045-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.243.126:38397
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab
--principal=kudu/127.3.243.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.126:38397
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.243.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781894810715282-4045-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:52:31.195596 26319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:52:31.197245 26319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:52:31.210035 26324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:52:31.210009 26325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:52:31.210237 26327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:52:31.210912 26319 server_base.cc:1034] running on GCE node
I20260619 18:52:31.211774 26319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:52:31.213367 26319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:52:31.214572 26319 hybrid_clock.cc:648] HybridClock initialized: now 1781895151214522 us; error 40 us; skew 500 ppm
I20260619 18:52:31.214927 26319 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:52:31 dist-test-slave-m9wg krb5kdc[26306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895151, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:52:31.223397 26319 init.cc:377] Logged in from keytab as kudu/127.3.243.126@KRBTEST.COM (short username kudu)
I20260619 18:52:31.225395 26319 webserver.cc:460] Webserver started at http://127.3.243.126:39571/ using document root <none> and password file <none>
I20260619 18:52:31.226250 26319 fs_manager.cc:359] Metadata directory not provided
I20260619 18:52:31.226383 26319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:52:31.226758 26319 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:52:31.230532 26319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781894810715282-4045-0/minicluster-data/master-0/data/instance:
uuid: "b2f8753174c04d428fa81fbf48432af7"
format_stamp: "Formatted at 2026-06-19 18:52:31 on dist-test-slave-m9wg"
server_key: "c6262f74d732cc6d887b3b6e7372b859"
server_key_iv: "f0a05391a6329b72dd275d4fc0b7f564"
server_key_version: "encryptionkey@0"
I20260619 18:52:31.231508 26319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781894810715282-4045-0/minicluster-data/master-0/wal/instance:
uuid: "b2f8753174c04d428fa81fbf48432af7"
format_stamp: "Formatted at 2026-06-19 18:52:31 on dist-test-slave-m9wg"
server_key: "c6262f74d732cc6d887b3b6e7372b859"
server_key_iv: "f0a05391a6329b72dd275d4fc0b7f564"
server_key_version: "encryptionkey@0"
I20260619 18:52:31.237973 26319 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.003s
I20260619 18:52:31.242223 26334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:52:31.244005 26319 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260619 18:52:31.244266 26319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781894810715282-4045-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781894810715282-4045-0/minicluster-data/master-0/wal
uuid: "b2f8753174c04d428fa81fbf48432af7"
format_stamp: "Formatted at 2026-06-19 18:52:31 on dist-test-slave-m9wg"
server_key: "c6262f74d732cc6d887b3b6e7372b859"
server_key_iv: "f0a05391a6329b72dd275d4fc0b7f564"
server_key_version: "encryptionkey@0"
I20260619 18:52:31.244588 26319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781894810715282-4045-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781894810715282-4045-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781894810715282-4045-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:52:31.252046 26319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:52:31.257463 26319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:52:31.257807 26319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:52:31.277971 26319 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.126:38397
I20260619 18:52:31.277992 26385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.126:38397 every 8 connection(s)
I20260619 18:52:31.279640 26319 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
I20260619 18:52:31.284183 26386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:52:31.287434 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 26319
I20260619 18:52:31.287622 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781894810715282-4045-0/minicluster-data/master-0/wal/instance
I20260619 18:52:31.288115 4045 external_mini_cluster.cc:1428] Setting key ec0c055efd18e647a251114459589273
I20260619 18:52:31.296754 26389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:31.290188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41235 (local address 127.3.243.126:38397)
0619 18:52:31.291030 (+ 842us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:31.291044 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:31.291094 (+ 50us) server_negotiation.cc:407] Connection header received
0619 18:52:31.292961 (+ 1867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:31.292989 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:31.293428 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:31.293851 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:31.295002 (+ 1151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:52:31.295008 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:52:31.295023 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:52:31.295038 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:52:31.295109 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:52:31.295134 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:52:31.295311 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:31.295429 (+ 118us) server_negotiation.cc:299] Negotiation successful
0619 18:52:31.295633 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":369,"threads_started":1}
I20260619 18:52:31.297154 26386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2f8753174c04d428fa81fbf48432af7: Bootstrap starting.
I20260619 18:52:31.300801 4045 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:52:31.301216 26386 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b2f8753174c04d428fa81fbf48432af7: Neither blocks nor log segments found. Creating new log.
I20260619 18:52:31.302335 26386 log.cc:824] T 00000000000000000000000000000000 P b2f8753174c04d428fa81fbf48432af7: Log is configured to *not* fsync() on all Append() calls
I20260619 18:52:31.305518 26386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2f8753174c04d428fa81fbf48432af7: No bootstrap required, opened a new log
I20260619 18:52:31.308137 26389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:31.305663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43187 (local address 127.3.243.126:38397)
0619 18:52:31.305845 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:31.305851 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:31.306277 (+ 426us) server_negotiation.cc:407] Connection header received
0619 18:52:31.306422 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:31.306428 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:31.306489 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:31.306592 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:31.307475 (+ 883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:52:31.307481 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:52:31.307486 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:52:31.307504 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:52:31.307557 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:52:31.307578 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:52:31.307692 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:31.307762 (+ 70us) server_negotiation.cc:299] Negotiation successful
0619 18:52:31.307826 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 18:52:31.313740 26386 raft_consensus.cc:348] T 00000000000000000000000000000000 P b2f8753174c04d428fa81fbf48432af7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2f8753174c04d428fa81fbf48432af7" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 38397 } }
I20260619 18:52:31.314141 26386 raft_consensus.cc:374] T 00000000000000000000000000000000 P b2f8753174c04d428fa81fbf48432af7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:52:31.314215 26386 raft_consensus.cc:729] T 00000000000000000000000000000000 P b2f8753174c04d428fa81fbf48432af7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2f8753174c04d428fa81fbf48432af7, State: Initialized, Role: FOLLOWER
I20260619 18:52:31.314808 26386 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2f8753174c04d428fa81fbf48432af7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2f8753174c04d428fa81fbf48432af7" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 38397 } }
I20260619 18:52:31.315037 26386 raft_consensus.cc:388] T 00000000000000000000000000000000 P b2f8753174c04d428fa81fbf48432af7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:52:31.315143 26386 raft_consensus.cc:482] T 00000000000000000000000000000000 P b2f8753174c04d428fa81fbf48432af7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:52:31.315268 26386 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b2f8753174c04d428fa81fbf48432af7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:52:31.317541 26386 raft_consensus.cc:504] T 00000000000000000000000000000000 P b2f8753174c04d428fa81fbf48432af7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2f8753174c04d428fa81fbf48432af7" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 38397 } }
I20260619 18:52:31.318039 26386 leader_election.cc:302] T 00000000000000000000000000000000 P b2f8753174c04d428fa81fbf48432af7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b2f8753174c04d428fa81fbf48432af7; no voters:
I20260619 18:52:31.318594 26386 leader_election.cc:288] T 00000000000000000000000000000000 P b2f8753174c04d428fa81fbf48432af7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:52:31.318714 26396 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b2f8753174c04d428fa81fbf48432af7 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:52:31.319726 26396 raft_consensus.cc:686] T 00000000000000000000000000000000 P b2f8753174c04d428fa81fbf48432af7 [term 1 LEADER]: Becoming Leader. State: Replica: b2f8753174c04d428fa81fbf48432af7, State: Running, Role: LEADER
I20260619 18:52:31.320312 26396 consensus_queue.cc:237] T 00000000000000000000000000000000 P b2f8753174c04d428fa81fbf48432af7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2f8753174c04d428fa81fbf48432af7" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 38397 } }
I20260619 18:52:31.320955 26386 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2f8753174c04d428fa81fbf48432af7 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:52:31.323623 26397 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2f8753174c04d428fa81fbf48432af7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b2f8753174c04d428fa81fbf48432af7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2f8753174c04d428fa81fbf48432af7" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 38397 } } }
I20260619 18:52:31.324091 26397 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2f8753174c04d428fa81fbf48432af7 [sys.catalog]: This master's current role is: LEADER
I20260619 18:52:31.323756 26398 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2f8753174c04d428fa81fbf48432af7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2f8753174c04d428fa81fbf48432af7. Latest consensus state: current_term: 1 leader_uuid: "b2f8753174c04d428fa81fbf48432af7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2f8753174c04d428fa81fbf48432af7" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 38397 } } }
I20260619 18:52:31.324582 26398 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2f8753174c04d428fa81fbf48432af7 [sys.catalog]: This master's current role is: LEADER
I20260619 18:52:31.325186 26404 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:52:31.332109 26404 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:52:31.341681 26404 catalog_manager.cc:1347] Generated new cluster ID: 13c8c1a4755f4b28956f395c01dfbe8d
I20260619 18:52:31.341794 26404 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:52:31.377027 26404 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:52:31.377525 26404 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:52:31.385048 26404 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b2f8753174c04d428fa81fbf48432af7: Generated new TSK 0
I20260619 18:52:31.385943 26404 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:52:31.403012 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 26319
W20260619 18:52:31.404271 26381 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:41235
W20260619 18:52:31.405036 4045 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 26319: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 18:52:31.405179 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 26319
2026-06-19T18:52:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1980 ms)
[----------] 18 tests from AuthTokenIssuingTest (35671 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894810715282-4045-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:52:31 dist-test-slave-m9wg krb5kdc[26417](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:52:31 dist-test-slave-m9wg krb5kdc[26417](info): set up 2 sockets
Jun 19 18:52:31 dist-test-slave-m9wg krb5kdc[26417](info): commencing operation
krb5kdc: starting...
W20260619 18:52:32.436884 4045 mini_kdc.cc:121] Time spent starting KDC: real 0.995s user 0.007s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894810715282-4045-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894810715282-4045-0/krb5kdc/test-user.keytab.
Jun 19 18:52:32 dist-test-slave-m9wg krb5kdc[26417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895152, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:52:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:52:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.243.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab.
Entry for principal kudu/127.3.243.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab.
I20260619 18:52:32.669799 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894810715282-4045-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894810715282-4045-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894810715282-4045-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.126:43087
--webserver_interface=127.3.243.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab
--principal=kudu/127.3.243.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:33621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.126:43087,127.3.243.125:36419,127.3.243.124:43433
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:52:33.013661 26430 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 18:52:33.013988 26430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:52:33.014086 26430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:52:33.014194 26430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:52:33.021178 26430 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:52:33.021260 26430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:52:33.021351 26430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:52:33.021415 26430 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:52:33.021498 26430 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:52:33.030925 26430 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:33621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894810715282-4045-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894810715282-4045-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.243.126:43087,127.3.243.125:36419,127.3.243.124:43433
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab
--principal=kudu/127.3.243.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.126:43087
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.243.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894810715282-4045-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:52:33.032193 26430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:52:33.033501 26430 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:52:33.044760 26435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:52:33.044699 26436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:52:33.044735 26438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:52:33.045836 26430 server_base.cc:1034] running on GCE node
I20260619 18:52:33.046541 26430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:52:33.047832 26430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:52:33.049041 26430 hybrid_clock.cc:648] HybridClock initialized: now 1781895153048984 us; error 38 us; skew 500 ppm
I20260619 18:52:33.049355 26430 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:52:33 dist-test-slave-m9wg krb5kdc[26417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895153, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:52:33.052304 26430 init.cc:377] Logged in from keytab as kudu/127.3.243.126@KRBTEST.COM (short username kudu)
I20260619 18:52:33.053745 26430 webserver.cc:460] Webserver started at http://127.3.243.126:46801/ using document root <none> and password file <none>
I20260619 18:52:33.054442 26430 fs_manager.cc:359] Metadata directory not provided
I20260619 18:52:33.054539 26430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:52:33.054837 26430 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:52:33.058198 26430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894810715282-4045-0/minicluster-data/master-0/data/instance:
uuid: "716297d21b414fec897a3c165d669318"
format_stamp: "Formatted at 2026-06-19 18:52:33 on dist-test-slave-m9wg"
server_key: "a838f8991c69702c678461469a7cafbb"
server_key_iv: "2e4f4511db15813f497160c16d8904cf"
server_key_version: "encryptionkey@0"
I20260619 18:52:33.058954 26430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894810715282-4045-0/minicluster-data/master-0/wal/instance:
uuid: "716297d21b414fec897a3c165d669318"
format_stamp: "Formatted at 2026-06-19 18:52:33 on dist-test-slave-m9wg"
server_key: "a838f8991c69702c678461469a7cafbb"
server_key_iv: "2e4f4511db15813f497160c16d8904cf"
server_key_version: "encryptionkey@0"
I20260619 18:52:33.064079 26430 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 18:52:33.067466 26445 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:52:33.068761 26430 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 18:52:33.068908 26430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894810715282-4045-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894810715282-4045-0/minicluster-data/master-0/wal
uuid: "716297d21b414fec897a3c165d669318"
format_stamp: "Formatted at 2026-06-19 18:52:33 on dist-test-slave-m9wg"
server_key: "a838f8991c69702c678461469a7cafbb"
server_key_iv: "2e4f4511db15813f497160c16d8904cf"
server_key_version: "encryptionkey@0"
I20260619 18:52:33.069093 26430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894810715282-4045-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894810715282-4045-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894810715282-4045-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:52:33.081720 26430 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:52:33.085565 26430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:52:33.085841 26430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:52:33.102993 26430 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.126:43087
I20260619 18:52:33.103012 26496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.126:43087 every 8 connection(s)
I20260619 18:52:33.104341 26430 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
I20260619 18:52:33.107164 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 26430
I20260619 18:52:33.107384 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894810715282-4045-0/minicluster-data/master-0/wal/instance
I20260619 18:52:33.108210 4045 external_mini_cluster.cc:1428] Setting key 8212d2b336435a064dae4b6cb0568591
I20260619 18:52:33.108650 26497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:52:33.117331 26497 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.243.126" port: 43087 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 18:52:33 dist-test-slave-m9wg krb5kdc[26417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895153, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.126@KRBTEST.COM for kudu/127.3.243.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.243.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.125@KRBTEST.COM" created.
I20260619 18:52:33.139424 26501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:33.120959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55216 (local address 127.3.243.126:43087)
0619 18:52:33.122580 (+ 1621us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:33.122588 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:33.122637 (+ 49us) server_negotiation.cc:407] Connection header received
0619 18:52:33.124108 (+ 1471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:33.124114 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:33.124388 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:33.124562 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:33.125807 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:33.127001 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:33.128382 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:33.128926 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:33.129095 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:33.132435 (+ 3340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:52:33.132443 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:52:33.132455 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:52:33.132484 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:52:33.135200 (+ 2716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:33.135955 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:33.135961 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:33.135963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:33.136116 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:33.136607 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:33.136611 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:33.136613 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:33.137163 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:52:33.137318 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:33.138058 (+ 740us) server_negotiation.cc:299] Negotiation successful
0619 18:52:33.138325 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":1413,"thread_start_us":1200,"threads_started":1}
I20260619 18:52:33.139477 26500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:33.120959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:43087 (local address 127.0.0.1:55216)
0619 18:52:33.121775 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0619 18:52:33.121820 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 18:52:33.123755 (+ 1935us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:52:33.124603 (+ 848us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:52:33.124609 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:52:33.125282 (+ 673us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:52:33.125617 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:52:33.125635 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:52:33.127141 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:52:33.127145 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:52:33.128261 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:52:33.128274 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:52:33.129147 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:52:33.129153 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:52:33.129390 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:33.130067 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:52:33.130096 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:52:33.132238 (+ 2142us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:52:33.135360 (+ 3122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:52:33.135372 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:52:33.135385 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:52:33.135823 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:52:33.136252 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:52:33.136258 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:52:33.136261 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:52:33.136488 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:52:33.137369 (+ 881us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:52:33.137377 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:52:33.137646 (+ 269us) client_negotiation.cc:769] Sending connection context
0619 18:52:33.137920 (+ 274us) client_negotiation.cc:240] Negotiation successful
0619 18:52:33.138501 (+ 581us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"mutex_wait_us":125,"thread_start_us":324,"threads_started":1}
I20260619 18:52:33.142592 26497 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.243.125" port: 36419 } has no permanent_uuid. Determining permanent_uuid...
I20260619 18:52:33.143736 26500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:33.143338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.125:36419 (local address 127.0.0.1:41716)
0619 18:52:33.143498 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 18:52:33.143612 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.125:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260619 18:52:33.144111 26449 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.243.125:36419: connect: Connection refused (error 111)
I20260619 18:52:33.145685 26500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:33.145328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.125:36419 (local address 127.0.0.1:41724)
0619 18:52:33.145457 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0619 18:52:33.145533 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.125:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260619 18:52:33.145982 26497 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.243.125:36419: Network error: Client connection negotiation failed: client connection to 127.3.243.125:36419: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.125.keytab.
Entry for principal kudu/127.3.243.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.125.keytab.
I20260619 18:52:33.167341 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894810715282-4045-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894810715282-4045-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894810715282-4045-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894810715282-4045-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.125:36419
--webserver_interface=127.3.243.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.125.keytab
--principal=kudu/127.3.243.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:33621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.126:43087,127.3.243.125:36419,127.3.243.124:43433
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 18:52:33.195348 26497 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.243.125" port: 36419 } attempt: 1
I20260619 18:52:33.196513 26500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:33.196035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.125:36419 (local address 127.0.0.1:41730)
0619 18:52:33.196245 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0619 18:52:33.196375 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.125:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260619 18:52:33.197451 26500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:33.197116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.125:36419 (local address 127.0.0.1:41742)
0619 18:52:33.197250 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0619 18:52:33.197325 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.125:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260619 18:52:33.197613 26497 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.243.125:36419: Network error: Client connection negotiation failed: client connection to 127.3.243.125:36419: connect: Connection refused (error 111)
I20260619 18:52:33.265894 26497 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.243.125" port: 36419 } attempt: 2
I20260619 18:52:33.267012 26500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:33.266541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.125:36419 (local address 127.0.0.1:41752)
0619 18:52:33.266784 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0619 18:52:33.266875 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.125:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260619 18:52:33.267921 26500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:33.267579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.125:36419 (local address 127.0.0.1:41760)
0619 18:52:33.267712 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0619 18:52:33.267798 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.125:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260619 18:52:33.268095 26497 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.243.125:36419: Network error: Client connection negotiation failed: client connection to 127.3.243.125:36419: connect: Connection refused (error 111)
I20260619 18:52:33.359382 26497 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.243.125" port: 36419 } attempt: 3
I20260619 18:52:33.360460 26500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:33.360013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.125:36419 (local address 127.0.0.1:41772)
0619 18:52:33.360233 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0619 18:52:33.360325 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.125:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260619 18:52:33.361392 26500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:33.361052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.125:36419 (local address 127.0.0.1:41774)
0619 18:52:33.361194 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0619 18:52:33.361259 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.125:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 18:52:33.361562 26497 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.243.125:36419: Network error: Client connection negotiation failed: client connection to 127.3.243.125:36419: connect: Connection refused (error 111)
W20260619 18:52:33.479306 26504 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 18:52:33.479620 26504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:52:33.479714 26504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:52:33.479830 26504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:52:33.486816 26504 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:52:33.486902 26504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:52:33.486984 26504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:52:33.487046 26504 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:52:33.487128 26504 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:52:33.496541 26504 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:33621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894810715282-4045-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894810715282-4045-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.243.126:43087,127.3.243.125:36419,127.3.243.124:43433
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.125.keytab
--principal=kudu/127.3.243.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.125:36419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894810715282-4045-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.243.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894810715282-4045-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:52:33.497695 26504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:52:33.498994 26504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 18:52:33.504772 26497 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.243.125" port: 36419 } attempt: 4
I20260619 18:52:33.505898 26500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:33.505396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.125:36419 (local address 127.0.0.1:41778)
0619 18:52:33.505669 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0619 18:52:33.505755 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.125:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141}
I20260619 18:52:33.506789 26500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:33.506450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.125:36419 (local address 127.0.0.1:41794)
0619 18:52:33.506592 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0619 18:52:33.506672 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.125:36419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260619 18:52:33.506979 26497 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.243.125:36419: Network error: Client connection negotiation failed: client connection to 127.3.243.125:36419: connect: Connection refused (error 111)
W20260619 18:52:33.510191 26509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:52:33.510231 26510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:52:33.510154 26512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:52:33.511255 26504 server_base.cc:1034] running on GCE node
I20260619 18:52:33.512096 26504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:52:33.513419 26504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:52:33.514635 26504 hybrid_clock.cc:648] HybridClock initialized: now 1781895153514571 us; error 39 us; skew 500 ppm
I20260619 18:52:33.514948 26504 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:52:33 dist-test-slave-m9wg krb5kdc[26417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895153, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:52:33.517892 26504 init.cc:377] Logged in from keytab as kudu/127.3.243.125@KRBTEST.COM (short username kudu)
I20260619 18:52:33.519235 26504 webserver.cc:460] Webserver started at http://127.3.243.125:44785/ using document root <none> and password file <none>
I20260619 18:52:33.519855 26504 fs_manager.cc:359] Metadata directory not provided
I20260619 18:52:33.519951 26504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:52:33.520288 26504 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:52:33.523012 26504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894810715282-4045-0/minicluster-data/master-1/data/instance:
uuid: "2e6bbc469c1c4f95bb2f58f5117bd358"
format_stamp: "Formatted at 2026-06-19 18:52:33 on dist-test-slave-m9wg"
server_key: "ff62751b3276b2b23b6dcfff4c273ffc"
server_key_iv: "0b03f03f66a139a235b28b0a15645ad0"
server_key_version: "encryptionkey@0"
I20260619 18:52:33.523725 26504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894810715282-4045-0/minicluster-data/master-1/wal/instance:
uuid: "2e6bbc469c1c4f95bb2f58f5117bd358"
format_stamp: "Formatted at 2026-06-19 18:52:33 on dist-test-slave-m9wg"
server_key: "ff62751b3276b2b23b6dcfff4c273ffc"
server_key_iv: "0b03f03f66a139a235b28b0a15645ad0"
server_key_version: "encryptionkey@0"
I20260619 18:52:33.528403 26504 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20260619 18:52:33.531605 26519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:52:33.532896 26504 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:52:33.533047 26504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894810715282-4045-0/minicluster-data/master-1/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894810715282-4045-0/minicluster-data/master-1/wal
uuid: "2e6bbc469c1c4f95bb2f58f5117bd358"
format_stamp: "Formatted at 2026-06-19 18:52:33 on dist-test-slave-m9wg"
server_key: "ff62751b3276b2b23b6dcfff4c273ffc"
server_key_iv: "0b03f03f66a139a235b28b0a15645ad0"
server_key_version: "encryptionkey@0"
I20260619 18:52:33.533226 26504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894810715282-4045-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894810715282-4045-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894810715282-4045-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:52:33.545670 26504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:52:33.549543 26504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:52:33.549785 26504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:52:33.566584 26504 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.125:36419
I20260619 18:52:33.566606 26570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.125:36419 every 8 connection(s)
I20260619 18:52:33.567857 26504 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894810715282-4045-0/minicluster-data/master-1/data/info.pb
I20260619 18:52:33.569970 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 26504
I20260619 18:52:33.570123 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894810715282-4045-0/minicluster-data/master-1/wal/instance
I20260619 18:52:33.570531 4045 external_mini_cluster.cc:1428] Setting key d5485f31185c98981147e5d5660d15d6
I20260619 18:52:33.571398 26571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:52:33.577706 26571 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.243.126" port: 43087 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 18:52:33 dist-test-slave-m9wg krb5kdc[26417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895153, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.125@KRBTEST.COM for kudu/127.3.243.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.243.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.124@KRBTEST.COM" created.
I20260619 18:52:33.596339 26501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:33.580124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55226 (local address 127.3.243.126:43087)
0619 18:52:33.580330 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:33.580336 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:33.581327 (+ 991us) server_negotiation.cc:407] Connection header received
0619 18:52:33.583079 (+ 1752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:33.583085 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:33.583163 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:33.583280 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:33.584667 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:33.585599 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:33.587091 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:33.587730 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:33.587860 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:33.591007 (+ 3147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:52:33.591014 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:52:33.591019 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:52:33.591060 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:52:33.593539 (+ 2479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:33.594319 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:33.594323 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:33.594325 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:33.594436 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:33.594813 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:33.594816 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:33.594818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:33.595072 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:52:33.595178 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:33.595901 (+ 723us) server_negotiation.cc:299] Negotiation successful
0619 18:52:33.596062 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 18:52:33.597430 26574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:33.580466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:43087 (local address 127.0.0.1:55226)
0619 18:52:33.581168 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0619 18:52:33.581216 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 18:52:33.582847 (+ 1631us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:52:33.583427 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:52:33.583434 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:52:33.583960 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:52:33.584517 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:52:33.584533 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:52:33.585777 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:52:33.585811 (+ 34us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:52:33.586947 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:52:33.586959 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:52:33.587972 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:52:33.587978 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:52:33.588188 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:33.588818 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:52:33.588837 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:52:33.590776 (+ 1939us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:52:33.593666 (+ 2890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:52:33.593675 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:52:33.593688 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:52:33.594211 (+ 523us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:52:33.594521 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:52:33.594525 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:52:33.594529 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:52:33.594722 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:52:33.595182 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:52:33.595189 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:52:33.595545 (+ 356us) client_negotiation.cc:769] Sending connection context
0619 18:52:33.595795 (+ 250us) client_negotiation.cc:240] Negotiation successful
0619 18:52:33.596236 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":291,"threads_started":1}
I20260619 18:52:33.599048 26571 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.243.125" port: 36419 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 18:52:33 dist-test-slave-m9wg krb5kdc[26417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895153, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.125@KRBTEST.COM for kudu/127.3.243.125@KRBTEST.COM
I20260619 18:52:33.612573 26574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:33.599733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.125:36419 (local address 127.0.0.1:41798)
0619 18:52:33.599931 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0619 18:52:33.599949 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 18:52:33.600042 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:52:33.600980 (+ 938us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:52:33.600983 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:52:33.601406 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:52:33.601566 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:52:33.601576 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:52:33.602900 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:52:33.602904 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:52:33.603711 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:52:33.603722 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:52:33.604732 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:52:33.604736 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:52:33.604871 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:33.605369 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:52:33.605384 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:52:33.607447 (+ 2063us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:52:33.610198 (+ 2751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:52:33.610203 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:52:33.610206 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:52:33.610625 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:52:33.610947 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:52:33.610952 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:52:33.610956 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:52:33.611117 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:52:33.611637 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:52:33.611642 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:52:33.611777 (+ 135us) client_negotiation.cc:769] Sending connection context
0619 18:52:33.611936 (+ 159us) client_negotiation.cc:240] Negotiation successful
0619 18:52:33.612211 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260619 18:52:33.612599 26575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:33.599934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41798 (local address 127.3.243.125:36419)
0619 18:52:33.600542 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:33.600549 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:33.600577 (+ 28us) server_negotiation.cc:407] Connection header received
0619 18:52:33.600623 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:33.600630 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:33.600828 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:33.600958 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:33.601739 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:33.602780 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:33.603860 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:33.604596 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:33.604704 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:33.607624 (+ 2920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:52:33.607634 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:52:33.607646 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:52:33.607672 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:52:33.610079 (+ 2407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:33.610732 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:33.610738 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:33.610740 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:33.610848 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:33.611208 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:33.611211 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:33.611213 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:33.611510 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:52:33.611645 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:33.612061 (+ 416us) server_negotiation.cc:299] Negotiation successful
0619 18:52:33.612313 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":60,"server-negotiator.queue_time_us":459,"thread_start_us":324,"threads_started":1}
I20260619 18:52:33.614936 26571 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.243.124" port: 43433 } has no permanent_uuid. Determining permanent_uuid...
I20260619 18:52:33.615890 26574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:33.615504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.124:43433 (local address 127.0.0.1:34162)
0619 18:52:33.615645 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0619 18:52:33.615755 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.124:43433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260619 18:52:33.616091 26520 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.243.124:43433: connect: Connection refused (error 111)
I20260619 18:52:33.617650 26574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:33.617239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.124:43433 (local address 127.0.0.1:34164)
0619 18:52:33.617436 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0619 18:52:33.617527 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.124:43433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260619 18:52:33.617914 26571 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.243.124:43433: Network error: Client connection negotiation failed: client connection to 127.3.243.124:43433: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.124.keytab.
Entry for principal kudu/127.3.243.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.124.keytab.
I20260619 18:52:33.623255 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894810715282-4045-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894810715282-4045-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894810715282-4045-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894810715282-4045-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.124:43433
--webserver_interface=127.3.243.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.124.keytab
--principal=kudu/127.3.243.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:33621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.126:43087,127.3.243.125:36419,127.3.243.124:43433
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 18:52:33.667217 26571 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.243.124" port: 43433 } attempt: 1
I20260619 18:52:33.668357 26574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:33.667873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.124:43433 (local address 127.0.0.1:34178)
0619 18:52:33.668061 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0619 18:52:33.668194 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.124:43433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260619 18:52:33.669432 26574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:33.668991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.124:43433 (local address 127.0.0.1:34182)
0619 18:52:33.669177 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0619 18:52:33.669285 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.124:43433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260619 18:52:33.669705 26571 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.243.124:43433: Network error: Client connection negotiation failed: client connection to 127.3.243.124:43433: connect: Connection refused (error 111)
I20260619 18:52:33.737931 26571 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.243.124" port: 43433 } attempt: 2
I20260619 18:52:33.739168 26574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:33.738609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.124:43433 (local address 127.0.0.1:34188)
0619 18:52:33.738837 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0619 18:52:33.738962 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.124:43433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260619 18:52:33.740315 26574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:33.739835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.124:43433 (local address 127.0.0.1:34194)
0619 18:52:33.740027 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0619 18:52:33.740122 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.124:43433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260619 18:52:33.740538 26571 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.243.124:43433: Network error: Client connection negotiation failed: client connection to 127.3.243.124:43433: connect: Connection refused (error 111)
I20260619 18:52:33.806241 26497 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.243.125" port: 36419 } attempt: 5
Jun 19 18:52:33 dist-test-slave-m9wg krb5kdc[26417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895153, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.126@KRBTEST.COM for kudu/127.3.243.125@KRBTEST.COM
I20260619 18:52:33.819473 26575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:33.807224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41812 (local address 127.3.243.125:36419)
0619 18:52:33.807436 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:33.807441 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:33.807457 (+ 16us) server_negotiation.cc:407] Connection header received
0619 18:52:33.807508 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:33.807511 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:33.807569 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:33.807652 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:33.808562 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:33.809255 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:33.810397 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:33.810829 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:33.810943 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:33.814458 (+ 3515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:52:33.814463 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:52:33.814466 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:52:33.814498 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:52:33.816913 (+ 2415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:33.817694 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:33.817698 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:33.817700 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:33.817816 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:33.818222 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:33.818226 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:33.818228 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:33.818527 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:52:33.818644 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:33.819068 (+ 424us) server_negotiation.cc:299] Negotiation successful
0619 18:52:33.819237 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 18:52:33.819651 26500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:33.806953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.125:36419 (local address 127.0.0.1:41812)
0619 18:52:33.807133 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0619 18:52:33.807154 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 18:52:33.807261 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:52:33.807725 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:52:33.807731 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:52:33.808203 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:52:33.808404 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:52:33.808414 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:52:33.809393 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:52:33.809396 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:52:33.810243 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:52:33.810256 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:52:33.810972 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:52:33.810979 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:52:33.811184 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:33.811831 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:52:33.811847 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:52:33.814217 (+ 2370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:52:33.817041 (+ 2824us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:52:33.817046 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:52:33.817051 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:52:33.817586 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:52:33.817915 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:52:33.817920 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:52:33.817923 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:52:33.818091 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:52:33.818637 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:52:33.818642 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:52:33.818812 (+ 170us) client_negotiation.cc:769] Sending connection context
0619 18:52:33.818963 (+ 151us) client_negotiation.cc:240] Negotiation successful
0619 18:52:33.819277 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260619 18:52:33.820658 26497 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.243.124" port: 43433 } has no permanent_uuid. Determining permanent_uuid...
I20260619 18:52:33.821696 26500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:33.821344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.124:43433 (local address 127.0.0.1:34202)
0619 18:52:33.821494 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0619 18:52:33.821568 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.124:43433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260619 18:52:33.822603 26500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:33.822310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.124:43433 (local address 127.0.0.1:34216)
0619 18:52:33.822431 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0619 18:52:33.822512 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.124:43433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260619 18:52:33.822814 26497 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.243.124:43433: Network error: Client connection negotiation failed: client connection to 127.3.243.124:43433: connect: Connection refused (error 111)
I20260619 18:52:33.831758 26571 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.243.124" port: 43433 } attempt: 3
I20260619 18:52:33.832849 26574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:33.832364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.124:43433 (local address 127.0.0.1:34232)
0619 18:52:33.832574 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0619 18:52:33.832678 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.124:43433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260619 18:52:33.833911 26574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:33.833444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.124:43433 (local address 127.0.0.1:34244)
0619 18:52:33.833670 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0619 18:52:33.833776 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.124:43433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
W20260619 18:52:33.834160 26571 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.243.124:43433: Network error: Client connection negotiation failed: client connection to 127.3.243.124:43433: connect: Connection refused (error 111)
I20260619 18:52:33.874027 26497 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.243.124" port: 43433 } attempt: 1
I20260619 18:52:33.875203 26500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:33.874758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.124:43433 (local address 127.0.0.1:34258)
0619 18:52:33.874948 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0619 18:52:33.875040 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.124:43433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260619 18:52:33.876187 26500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:33.875839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.124:43433 (local address 127.0.0.1:34270)
0619 18:52:33.875986 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0619 18:52:33.876057 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.124:43433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260619 18:52:33.876389 26497 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.243.124:43433: Network error: Client connection negotiation failed: client connection to 127.3.243.124:43433: connect: Connection refused (error 111)
W20260619 18:52:33.935896 26578 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 18:52:33.936232 26578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:52:33.936324 26578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:52:33.936439 26578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:52:33.943560 26578 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:52:33.943645 26578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:52:33.943703 26578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:52:33.943770 26578 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:52:33.943847 26578 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:52:33.944574 26497 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.243.124" port: 43433 } attempt: 2
I20260619 18:52:33.945704 26500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:33.945234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.124:43433 (local address 127.0.0.1:34280)
0619 18:52:33.945454 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0619 18:52:33.945546 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.124:43433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260619 18:52:33.946712 26500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:33.946389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.124:43433 (local address 127.0.0.1:34294)
0619 18:52:33.946526 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0619 18:52:33.946596 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.124:43433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260619 18:52:33.946893 26497 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.243.124:43433: Network error: Client connection negotiation failed: client connection to 127.3.243.124:43433: connect: Connection refused (error 111)
I20260619 18:52:33.954190 26578 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:33621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894810715282-4045-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894810715282-4045-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.243.126:43087,127.3.243.125:36419,127.3.243.124:43433
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.124.keytab
--principal=kudu/127.3.243.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.124:43433
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894810715282-4045-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.243.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894810715282-4045-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:52:33.955438 26578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:52:33.956799 26578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:52:33.972623 26584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:52:33.972813 26586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:52:33.973022 26583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:52:33.974021 26578 server_base.cc:1034] running on GCE node
I20260619 18:52:33.974776 26578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:52:33.976212 26578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:52:33.977448 26578 hybrid_clock.cc:648] HybridClock initialized: now 1781895153977379 us; error 49 us; skew 500 ppm
I20260619 18:52:33.977380 26571 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.243.124" port: 43433 } attempt: 4
I20260619 18:52:33.977811 26578 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 18:52:33.978523 26574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:33.977981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.124:43433 (local address 127.0.0.1:34302)
0619 18:52:33.978207 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0619 18:52:33.978308 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.124:43433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260619 18:52:33.979455 26574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:33.979109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.124:43433 (local address 127.0.0.1:34312)
0619 18:52:33.979250 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0619 18:52:33.979340 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.124:43433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260619 18:52:33.979642 26571 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.243.124:43433: Network error: Client connection negotiation failed: client connection to 127.3.243.124:43433: connect: Connection refused (error 111)
Jun 19 18:52:33 dist-test-slave-m9wg krb5kdc[26417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895153, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:52:33.981263 26578 init.cc:377] Logged in from keytab as kudu/127.3.243.124@KRBTEST.COM (short username kudu)
I20260619 18:52:33.982760 26578 webserver.cc:460] Webserver started at http://127.3.243.124:32921/ using document root <none> and password file <none>
I20260619 18:52:33.983394 26578 fs_manager.cc:359] Metadata directory not provided
I20260619 18:52:33.983486 26578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:52:33.983781 26578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:52:33.986718 26578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894810715282-4045-0/minicluster-data/master-2/data/instance:
uuid: "49f8ec6ad17d4f818f3f88b7a864b1df"
format_stamp: "Formatted at 2026-06-19 18:52:33 on dist-test-slave-m9wg"
server_key: "70701d3e023e1df550e1c308793df168"
server_key_iv: "b03e058a19b95a7c601629f1e8f75c8d"
server_key_version: "encryptionkey@0"
I20260619 18:52:33.987438 26578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894810715282-4045-0/minicluster-data/master-2/wal/instance:
uuid: "49f8ec6ad17d4f818f3f88b7a864b1df"
format_stamp: "Formatted at 2026-06-19 18:52:33 on dist-test-slave-m9wg"
server_key: "70701d3e023e1df550e1c308793df168"
server_key_iv: "b03e058a19b95a7c601629f1e8f75c8d"
server_key_version: "encryptionkey@0"
I20260619 18:52:33.992352 26578 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 18:52:33.995700 26593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:52:33.997049 26578 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:52:33.997195 26578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894810715282-4045-0/minicluster-data/master-2/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894810715282-4045-0/minicluster-data/master-2/wal
uuid: "49f8ec6ad17d4f818f3f88b7a864b1df"
format_stamp: "Formatted at 2026-06-19 18:52:33 on dist-test-slave-m9wg"
server_key: "70701d3e023e1df550e1c308793df168"
server_key_iv: "b03e058a19b95a7c601629f1e8f75c8d"
server_key_version: "encryptionkey@0"
I20260619 18:52:33.997390 26578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894810715282-4045-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894810715282-4045-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894810715282-4045-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:52:34.021979 26578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:52:34.026181 26578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:52:34.026427 26578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:52:34.043447 26578 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.124:43433
I20260619 18:52:34.043467 26644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.124:43433 every 8 connection(s)
I20260619 18:52:34.044859 26578 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894810715282-4045-0/minicluster-data/master-2/data/info.pb
I20260619 18:52:34.048491 26645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:52:34.053100 26497 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.243.124" port: 43433 } attempt: 3
I20260619 18:52:34.054989 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 26578
I20260619 18:52:34.055166 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894810715282-4045-0/minicluster-data/master-2/wal/instance
I20260619 18:52:34.055685 4045 external_mini_cluster.cc:1428] Setting key 5a5a3714281437df7acbe9225317db42
I20260619 18:52:34.055455 26645 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.243.126" port: 43087 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 18:52:34 dist-test-slave-m9wg krb5kdc[26417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895153, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.126@KRBTEST.COM for kudu/127.3.243.124@KRBTEST.COM
Jun 19 18:52:34 dist-test-slave-m9wg krb5kdc[26417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895153, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.124@KRBTEST.COM for kudu/127.3.243.126@KRBTEST.COM
I20260619 18:52:34.078676 26500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:34.053782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.124:43433 (local address 127.0.0.1:34318)
0619 18:52:34.053971 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0619 18:52:34.053992 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 18:52:34.054132 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:52:34.059980 (+ 5848us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:52:34.059987 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:52:34.060587 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:52:34.060835 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:52:34.060851 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:52:34.064028 (+ 3177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:52:34.064035 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:52:34.065655 (+ 1620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:52:34.065675 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:52:34.067172 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:52:34.067178 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:52:34.067401 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:34.068139 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:52:34.068183 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:52:34.071002 (+ 2819us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:52:34.075479 (+ 4477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:52:34.075484 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:52:34.075488 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:52:34.076003 (+ 515us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:52:34.076476 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:52:34.076481 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:52:34.076484 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:52:34.076644 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:52:34.077627 (+ 983us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:52:34.077633 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:52:34.077860 (+ 227us) client_negotiation.cc:769] Sending connection context
0619 18:52:34.078015 (+ 155us) client_negotiation.cc:240] Negotiation successful
0619 18:52:34.078283 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
Jun 19 18:52:34 dist-test-slave-m9wg krb5kdc[26417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895152, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.124@KRBTEST.COM
I20260619 18:52:34.080047 26647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:34.054735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34318 (local address 127.3.243.124:43433)
0619 18:52:34.055484 (+ 749us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:34.055496 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:34.055535 (+ 39us) server_negotiation.cc:407] Connection header received
0619 18:52:34.058811 (+ 3276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:34.058840 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:34.059516 (+ 676us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:34.061121 (+ 1605us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:34.061695 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:34.063803 (+ 2108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:34.065869 (+ 2066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:34.066982 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:34.069427 (+ 2445us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:34.071257 (+ 1830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:52:34.071267 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:52:34.071281 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:52:34.071325 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:52:34.075344 (+ 4019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:34.076174 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:34.076183 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:34.076191 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:34.076362 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:34.076766 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:34.076771 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:34.076773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:34.077463 (+ 690us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:52:34.077640 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:34.078305 (+ 665us) server_negotiation.cc:299] Negotiation successful
0619 18:52:34.078725 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":522,"thread_start_us":332,"threads_started":1}
I20260619 18:52:34.085745 26501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:34.059027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55242 (local address 127.3.243.126:43087)
0619 18:52:34.059208 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:34.059214 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:34.069955 (+ 10741us) server_negotiation.cc:407] Connection header received
0619 18:52:34.070235 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:34.070240 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:34.070310 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:34.070372 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:34.071718 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:34.072773 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:34.075358 (+ 2585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:34.075927 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:34.076038 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:34.079600 (+ 3562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:52:34.079606 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:52:34.079611 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:52:34.079646 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:52:34.082589 (+ 2943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:34.083398 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:34.083404 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:34.083407 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:34.083550 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:34.083983 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:34.083988 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:34.083990 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:34.084382 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:52:34.084547 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:34.085152 (+ 605us) server_negotiation.cc:299] Negotiation successful
0619 18:52:34.085369 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 18:52:34.086362 26648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:34.056646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34324 (local address 127.3.243.124:43433)
0619 18:52:34.069508 (+ 12862us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:34.069513 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:34.069540 (+ 27us) server_negotiation.cc:407] Connection header received
0619 18:52:34.069594 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:34.069598 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:34.069667 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:34.069786 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:34.071715 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:34.073028 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:34.074350 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:34.075091 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:34.075206 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:34.080417 (+ 5211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:52:34.080423 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:52:34.080428 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:52:34.080474 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:52:34.083600 (+ 3126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:34.084346 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:34.084351 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:34.084355 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:34.084465 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:34.084807 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:34.084812 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:34.084816 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:34.085076 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:52:34.085328 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:34.085596 (+ 268us) server_negotiation.cc:299] Negotiation successful
0619 18:52:34.085787 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":12693,"thread_start_us":12496,"threads_started":1}
I20260619 18:52:34.086760 26650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:34.058568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:43087 (local address 127.0.0.1:55242)
0619 18:52:34.069770 (+ 11202us) negotiation.cc:107] Waiting for socket to connect
0619 18:52:34.069828 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 18:52:34.070073 (+ 245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:52:34.070602 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:52:34.070609 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:52:34.071248 (+ 639us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:52:34.071591 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:52:34.071602 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:52:34.073170 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:52:34.073179 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:52:34.075160 (+ 1981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:52:34.075179 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:52:34.076076 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:52:34.076082 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:52:34.076294 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:34.076859 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:52:34.076881 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:52:34.079419 (+ 2538us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:52:34.082711 (+ 3292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:52:34.082722 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:52:34.082732 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:52:34.083288 (+ 556us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:52:34.083671 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:52:34.083676 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:52:34.083679 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:52:34.083861 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:52:34.084616 (+ 755us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:52:34.084623 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:52:34.084801 (+ 178us) client_negotiation.cc:769] Sending connection context
0619 18:52:34.086006 (+ 1205us) client_negotiation.cc:240] Negotiation successful
0619 18:52:34.086294 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10972,"mutex_wait_us":69,"spinlock_wait_cycles":22784,"thread_start_us":11306,"threads_started":1}
I20260619 18:52:34.088248 26497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 716297d21b414fec897a3c165d669318: Bootstrap starting.
I20260619 18:52:34.088306 26645 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.243.125" port: 36419 } has no permanent_uuid. Determining permanent_uuid...
I20260619 18:52:34.091931 26497 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 716297d21b414fec897a3c165d669318: Neither blocks nor log segments found. Creating new log.
I20260619 18:52:34.093232 26497 log.cc:824] T 00000000000000000000000000000000 P 716297d21b414fec897a3c165d669318: Log is configured to *not* fsync() on all Append() calls
Jun 19 18:52:34 dist-test-slave-m9wg krb5kdc[26417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895153, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.124@KRBTEST.COM for kudu/127.3.243.125@KRBTEST.COM
I20260619 18:52:34.096817 26497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 716297d21b414fec897a3c165d669318: No bootstrap required, opened a new log
Jun 19 18:52:34 dist-test-slave-m9wg krb5kdc[26417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895152, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.126@KRBTEST.COM
I20260619 18:52:34.103174 26575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:34.089411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41820 (local address 127.3.243.125:36419)
0619 18:52:34.089566 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:34.089570 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:34.089584 (+ 14us) server_negotiation.cc:407] Connection header received
0619 18:52:34.089630 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:34.089633 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:34.089693 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:34.089772 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:34.090535 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:34.091526 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:34.092729 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:34.093260 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:34.093385 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:34.096937 (+ 3552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:52:34.096944 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:52:34.096949 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:52:34.096989 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:52:34.100012 (+ 3023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:34.100824 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:34.100829 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:34.100833 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:34.101000 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:34.101522 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:34.101527 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:34.101531 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:34.102012 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:52:34.102158 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:34.102591 (+ 433us) server_negotiation.cc:299] Negotiation successful
0619 18:52:34.102812 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 18:52:34.103955 26650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:34.089048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.125:36419 (local address 127.0.0.1:41820)
0619 18:52:34.089213 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0619 18:52:34.089230 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 18:52:34.089326 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:52:34.089815 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:52:34.089820 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:52:34.090182 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:52:34.090355 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:52:34.090366 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:52:34.091677 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:52:34.091681 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:52:34.092593 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:52:34.092605 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:52:34.093476 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:52:34.093483 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:52:34.093656 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:34.094313 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:52:34.094336 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:52:34.096709 (+ 2373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:52:34.100170 (+ 3461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:52:34.100176 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:52:34.100180 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:52:34.100676 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:52:34.101117 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:52:34.101122 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:52:34.101125 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:52:34.101290 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:52:34.102131 (+ 841us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:52:34.102136 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:52:34.102291 (+ 155us) client_negotiation.cc:769] Sending connection context
0619 18:52:34.103347 (+ 1056us) client_negotiation.cc:240] Negotiation successful
0619 18:52:34.103633 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260619 18:52:34.105072 26645 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.243.124" port: 43433 } has no permanent_uuid. Determining permanent_uuid...
I20260619 18:52:34.105505 26501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:34.087969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55254 (local address 127.3.243.126:43087)
0619 18:52:34.088118 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:34.088123 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:34.088403 (+ 280us) server_negotiation.cc:407] Connection header received
0619 18:52:34.088611 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:34.088617 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:34.088683 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:34.088772 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:34.090164 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:34.091344 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:34.092976 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:34.093623 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:34.093727 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:34.098509 (+ 4782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:52:34.098515 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:52:34.098520 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:52:34.098564 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:52:34.102101 (+ 3537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:34.103020 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:34.103025 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:34.103028 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:34.103201 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:34.103710 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:34.103715 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:34.103718 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:34.104106 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:52:34.104414 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:34.104962 (+ 548us) server_negotiation.cc:299] Negotiation successful
0619 18:52:34.105170 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 18:52:34.106720 26497 raft_consensus.cc:348] T 00000000000000000000000000000000 P 716297d21b414fec897a3c165d669318 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716297d21b414fec897a3c165d669318" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 43087 } } peers { permanent_uuid: "2e6bbc469c1c4f95bb2f58f5117bd358" member_type: VOTER last_known_addr { host: "127.3.243.125" port: 36419 } } peers { permanent_uuid: "49f8ec6ad17d4f818f3f88b7a864b1df" member_type: VOTER last_known_addr { host: "127.3.243.124" port: 43433 } }
I20260619 18:52:34.107180 26497 raft_consensus.cc:729] T 00000000000000000000000000000000 P 716297d21b414fec897a3c165d669318 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 716297d21b414fec897a3c165d669318, State: Initialized, Role: FOLLOWER
I20260619 18:52:34.107733 26497 consensus_queue.cc:260] T 00000000000000000000000000000000 P 716297d21b414fec897a3c165d669318 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716297d21b414fec897a3c165d669318" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 43087 } } peers { permanent_uuid: "2e6bbc469c1c4f95bb2f58f5117bd358" member_type: VOTER last_known_addr { host: "127.3.243.125" port: 36419 } } peers { permanent_uuid: "49f8ec6ad17d4f818f3f88b7a864b1df" member_type: VOTER last_known_addr { host: "127.3.243.124" port: 43433 } }
Jun 19 18:52:34 dist-test-slave-m9wg krb5kdc[26417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895153, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.124@KRBTEST.COM for kudu/127.3.243.124@KRBTEST.COM
Jun 19 18:52:34 dist-test-slave-m9wg krb5kdc[26417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895152, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.125@KRBTEST.COM
I20260619 18:52:34.119401 26652 sys_catalog.cc:455] T 00000000000000000000000000000000 P 716297d21b414fec897a3c165d669318 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716297d21b414fec897a3c165d669318" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 43087 } } peers { permanent_uuid: "2e6bbc469c1c4f95bb2f58f5117bd358" member_type: VOTER last_known_addr { host: "127.3.243.125" port: 36419 } } peers { permanent_uuid: "49f8ec6ad17d4f818f3f88b7a864b1df" member_type: VOTER last_known_addr { host: "127.3.243.124" port: 43433 } } }
I20260619 18:52:34.119825 26652 sys_catalog.cc:458] T 00000000000000000000000000000000 P 716297d21b414fec897a3c165d669318 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:52:34.120820 26497 sys_catalog.cc:564] T 00000000000000000000000000000000 P 716297d21b414fec897a3c165d669318 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:52:34.121897 26650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:34.105764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.124:43433 (local address 127.0.0.1:34338)
0619 18:52:34.106032 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0619 18:52:34.106049 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 18:52:34.106302 (+ 253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:52:34.106724 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:52:34.106729 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:52:34.107143 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:52:34.107336 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:52:34.107348 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:52:34.108657 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:52:34.108662 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:52:34.110027 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:52:34.110040 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:52:34.111090 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:52:34.111095 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:52:34.111319 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:34.111912 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:52:34.111935 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:52:34.114941 (+ 3006us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:52:34.118801 (+ 3860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:52:34.118806 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:52:34.118810 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:52:34.119263 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:52:34.120315 (+ 1052us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:52:34.120319 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:52:34.120322 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:52:34.120441 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:52:34.121006 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:52:34.121011 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:52:34.121143 (+ 132us) client_negotiation.cc:769] Sending connection context
0619 18:52:34.121257 (+ 114us) client_negotiation.cc:240] Negotiation successful
0619 18:52:34.121537 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171}
I20260619 18:52:34.121958 26648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:34.106045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34338 (local address 127.3.243.124:43433)
0619 18:52:34.106222 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:34.106228 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:34.106246 (+ 18us) server_negotiation.cc:407] Connection header received
0619 18:52:34.106419 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:34.106424 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:34.106501 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:34.106559 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:34.107484 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:34.108504 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:34.110164 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:34.110801 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:34.110885 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:34.115112 (+ 4227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:52:34.115119 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:52:34.115123 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:52:34.115156 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:52:34.118578 (+ 3422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:34.119371 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:34.119376 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:34.119378 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:34.119522 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:34.120543 (+ 1021us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:34.120549 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:34.120552 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:34.120826 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:52:34.120974 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:34.121380 (+ 406us) server_negotiation.cc:299] Negotiation successful
0619 18:52:34.121675 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":105,"server-negotiator.queue_time_us":55}
I20260619 18:52:34.126190 26575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:34.107904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41828 (local address 127.3.243.125:36419)
0619 18:52:34.108037 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:34.108042 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:34.108277 (+ 235us) server_negotiation.cc:407] Connection header received
0619 18:52:34.108509 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:34.108514 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:34.108581 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:34.108648 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:34.110247 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:34.111305 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:34.113049 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:34.113690 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:34.113801 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:34.118998 (+ 5197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:52:34.119004 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:52:34.119009 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:52:34.119047 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:52:34.122328 (+ 3281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:34.123336 (+ 1008us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:34.123341 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:34.123344 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:34.123507 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:34.123966 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:34.123971 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:34.123974 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:34.124332 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:52:34.124575 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:34.125656 (+ 1081us) server_negotiation.cc:299] Negotiation successful
0619 18:52:34.125873 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 18:52:34.127710 26645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49f8ec6ad17d4f818f3f88b7a864b1df: Bootstrap starting.
I20260619 18:52:34.131335 26645 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 49f8ec6ad17d4f818f3f88b7a864b1df: Neither blocks nor log segments found. Creating new log.
I20260619 18:52:34.132597 26645 log.cc:824] T 00000000000000000000000000000000 P 49f8ec6ad17d4f818f3f88b7a864b1df: Log is configured to *not* fsync() on all Append() calls
I20260619 18:52:34.135260 26645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49f8ec6ad17d4f818f3f88b7a864b1df: No bootstrap required, opened a new log
I20260619 18:52:34.136108 4045 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:52:34.139811 26472 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "716297d21b414fec897a3c165d669318"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55254
I20260619 18:52:34.140244 26472 raft_consensus.cc:482] T 00000000000000000000000000000000 P 716297d21b414fec897a3c165d669318 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 18:52:34.140451 26472 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 716297d21b414fec897a3c165d669318 [term 0 FOLLOWER]: Advancing to term 1
W20260619 18:52:34.142275 26663 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 716297d21b414fec897a3c165d669318: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 18:52:34.142408 26663 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 18:52:34.143978 26472 raft_consensus.cc:504] T 00000000000000000000000000000000 P 716297d21b414fec897a3c165d669318 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716297d21b414fec897a3c165d669318" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 43087 } } peers { permanent_uuid: "2e6bbc469c1c4f95bb2f58f5117bd358" member_type: VOTER last_known_addr { host: "127.3.243.125" port: 36419 } } peers { permanent_uuid: "49f8ec6ad17d4f818f3f88b7a864b1df" member_type: VOTER last_known_addr { host: "127.3.243.124" port: 43433 } }
I20260619 18:52:34.131765 26430 master_runner.cc:186] Error getting master registration for 127.3.243.125:36419: OK, instance_id { permanent_uuid: "2e6bbc469c1c4f95bb2f58f5117bd358" instance_seqno: 1781895153561893 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 18:52:34.146463 26430 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 18:52:34.146950 26472 leader_election.cc:288] T 00000000000000000000000000000000 P 716297d21b414fec897a3c165d669318 [CANDIDATE]: Term 1 election: Requested vote from peers 2e6bbc469c1c4f95bb2f58f5117bd358 (127.3.243.125:36419), 49f8ec6ad17d4f818f3f88b7a864b1df (127.3.243.124:43433)
I20260619 18:52:34.146886 26645 raft_consensus.cc:348] T 00000000000000000000000000000000 P 49f8ec6ad17d4f818f3f88b7a864b1df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716297d21b414fec897a3c165d669318" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 43087 } } peers { permanent_uuid: "2e6bbc469c1c4f95bb2f58f5117bd358" member_type: VOTER last_known_addr { host: "127.3.243.125" port: 36419 } } peers { permanent_uuid: "49f8ec6ad17d4f818f3f88b7a864b1df" member_type: VOTER last_known_addr { host: "127.3.243.124" port: 43433 } }
I20260619 18:52:34.147327 26645 raft_consensus.cc:729] T 00000000000000000000000000000000 P 49f8ec6ad17d4f818f3f88b7a864b1df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49f8ec6ad17d4f818f3f88b7a864b1df, State: Initialized, Role: FOLLOWER
I20260619 18:52:34.147828 26645 consensus_queue.cc:260] T 00000000000000000000000000000000 P 49f8ec6ad17d4f818f3f88b7a864b1df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716297d21b414fec897a3c165d669318" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 43087 } } peers { permanent_uuid: "2e6bbc469c1c4f95bb2f58f5117bd358" member_type: VOTER last_known_addr { host: "127.3.243.125" port: 36419 } } peers { permanent_uuid: "49f8ec6ad17d4f818f3f88b7a864b1df" member_type: VOTER last_known_addr { host: "127.3.243.124" port: 43433 } }
I20260619 18:52:34.160797 26666 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49f8ec6ad17d4f818f3f88b7a864b1df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716297d21b414fec897a3c165d669318" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 43087 } } peers { permanent_uuid: "2e6bbc469c1c4f95bb2f58f5117bd358" member_type: VOTER last_known_addr { host: "127.3.243.125" port: 36419 } } peers { permanent_uuid: "49f8ec6ad17d4f818f3f88b7a864b1df" member_type: VOTER last_known_addr { host: "127.3.243.124" port: 43433 } } }
I20260619 18:52:34.161293 26666 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49f8ec6ad17d4f818f3f88b7a864b1df [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:52:34.162175 26645 sys_catalog.cc:564] T 00000000000000000000000000000000 P 49f8ec6ad17d4f818f3f88b7a864b1df [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:52:34.163025 26620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "716297d21b414fec897a3c165d669318" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "49f8ec6ad17d4f818f3f88b7a864b1df"
I20260619 18:52:34.163532 26620 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 49f8ec6ad17d4f818f3f88b7a864b1df [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:52:34.167342 26620 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 49f8ec6ad17d4f818f3f88b7a864b1df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 716297d21b414fec897a3c165d669318 in term 1.
I20260619 18:52:34.168221 26446 leader_election.cc:302] T 00000000000000000000000000000000 P 716297d21b414fec897a3c165d669318 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49f8ec6ad17d4f818f3f88b7a864b1df, 716297d21b414fec897a3c165d669318; no voters:
I20260619 18:52:34.169071 26652 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 716297d21b414fec897a3c165d669318 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:52:34.170341 26652 raft_consensus.cc:686] T 00000000000000000000000000000000 P 716297d21b414fec897a3c165d669318 [term 1 LEADER]: Becoming Leader. State: Replica: 716297d21b414fec897a3c165d669318, State: Running, Role: LEADER
I20260619 18:52:34.171018 26652 consensus_queue.cc:237] T 00000000000000000000000000000000 P 716297d21b414fec897a3c165d669318 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716297d21b414fec897a3c165d669318" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 43087 } } peers { permanent_uuid: "2e6bbc469c1c4f95bb2f58f5117bd358" member_type: VOTER last_known_addr { host: "127.3.243.125" port: 36419 } } peers { permanent_uuid: "49f8ec6ad17d4f818f3f88b7a864b1df" member_type: VOTER last_known_addr { host: "127.3.243.124" port: 43433 } }
I20260619 18:52:34.192891 26671 sys_catalog.cc:455] T 00000000000000000000000000000000 P 716297d21b414fec897a3c165d669318 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 716297d21b414fec897a3c165d669318. Latest consensus state: current_term: 1 leader_uuid: "716297d21b414fec897a3c165d669318" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716297d21b414fec897a3c165d669318" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 43087 } } peers { permanent_uuid: "2e6bbc469c1c4f95bb2f58f5117bd358" member_type: VOTER last_known_addr { host: "127.3.243.125" port: 36419 } } peers { permanent_uuid: "49f8ec6ad17d4f818f3f88b7a864b1df" member_type: VOTER last_known_addr { host: "127.3.243.124" port: 43433 } } }
I20260619 18:52:34.193325 26671 sys_catalog.cc:458] T 00000000000000000000000000000000 P 716297d21b414fec897a3c165d669318 [sys.catalog]: This master's current role is: LEADER
I20260619 18:52:34.194468 26684 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:52:34.201370 26684 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:52:34.221640 26575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:34.185635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41834 (local address 127.3.243.125:36419)
0619 18:52:34.185820 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:34.185827 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:34.189422 (+ 3595us) server_negotiation.cc:407] Connection header received
0619 18:52:34.189590 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:34.189595 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:34.189677 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:34.189763 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:34.195056 (+ 5293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:34.196171 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:34.205898 (+ 9727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:34.206523 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:34.206646 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:34.213314 (+ 6668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:52:34.213321 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:52:34.213326 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:52:34.213370 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:52:34.216968 (+ 3598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:34.218131 (+ 1163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:34.218136 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:34.218139 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:34.218366 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:34.219045 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:34.219050 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:34.219054 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:34.219618 (+ 564us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:52:34.219943 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:34.220753 (+ 810us) server_negotiation.cc:299] Negotiation successful
0619 18:52:34.221128 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 18:52:34.235317 26501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:34.176994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55270 (local address 127.3.243.126:43087)
0619 18:52:34.177188 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:34.177194 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:34.187171 (+ 9977us) server_negotiation.cc:407] Connection header received
0619 18:52:34.187415 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:34.187421 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:34.187521 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:34.187626 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:34.205948 (+ 18322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:34.207015 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:34.208840 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:34.209501 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:34.209622 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:34.219828 (+ 10206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:52:34.219835 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:52:34.219840 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:52:34.219895 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:52:34.223573 (+ 3678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:34.224481 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:34.224487 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:34.224490 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:34.224670 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:34.229193 (+ 4523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:34.229199 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:34.229203 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:34.229654 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:52:34.229915 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:34.231660 (+ 1745us) server_negotiation.cc:299] Negotiation successful
0619 18:52:34.234683 (+ 3023us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 18:52:34.238611 26648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:34.187788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34350 (local address 127.3.243.124:43433)
0619 18:52:34.187954 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:34.187959 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:34.190001 (+ 2042us) server_negotiation.cc:407] Connection header received
0619 18:52:34.191162 (+ 1161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:34.191168 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:34.191255 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:34.191313 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:34.197773 (+ 6460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:34.198768 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:34.210347 (+ 11579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:34.210978 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:34.211097 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:34.224285 (+ 13188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:52:34.224293 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:52:34.224298 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:52:34.224342 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:52:34.227741 (+ 3399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:34.228574 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:34.228580 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:34.228583 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:34.228749 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:34.230383 (+ 1634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:34.230388 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:34.230392 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:34.230801 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:52:34.231081 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:34.237927 (+ 6846us) server_negotiation.cc:299] Negotiation successful
0619 18:52:34.238216 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 18:52:34.246826 26620 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 49f8ec6ad17d4f818f3f88b7a864b1df [term 1 FOLLOWER]: Refusing update from remote peer 716297d21b414fec897a3c165d669318: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:52:34.248752 26652 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 716297d21b414fec897a3c165d669318 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49f8ec6ad17d4f818f3f88b7a864b1df" member_type: VOTER last_known_addr { host: "127.3.243.124" port: 43433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:52:34.251469 26688 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 18:52:34.257570 26578 master_runner.cc:186] Error getting master registration for 127.3.243.125:36419: OK, instance_id { permanent_uuid: "2e6bbc469c1c4f95bb2f58f5117bd358" instance_seqno: 1781895153561893 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 18:52:34.269730 26578 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 18:52:34.270375 26666 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49f8ec6ad17d4f818f3f88b7a864b1df [sys.catalog]: SysCatalogTable state changed. Reason: New leader 716297d21b414fec897a3c165d669318. Latest consensus state: current_term: 1 leader_uuid: "716297d21b414fec897a3c165d669318" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716297d21b414fec897a3c165d669318" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 43087 } } peers { permanent_uuid: "2e6bbc469c1c4f95bb2f58f5117bd358" member_type: VOTER last_known_addr { host: "127.3.243.125" port: 36419 } } peers { permanent_uuid: "49f8ec6ad17d4f818f3f88b7a864b1df" member_type: VOTER last_known_addr { host: "127.3.243.124" port: 43433 } } }
I20260619 18:52:34.270649 26666 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49f8ec6ad17d4f818f3f88b7a864b1df [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:52:34.275602 26652 sys_catalog.cc:455] T 00000000000000000000000000000000 P 716297d21b414fec897a3c165d669318 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "716297d21b414fec897a3c165d669318" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716297d21b414fec897a3c165d669318" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 43087 } } peers { permanent_uuid: "2e6bbc469c1c4f95bb2f58f5117bd358" member_type: VOTER last_known_addr { host: "127.3.243.125" port: 36419 } } peers { permanent_uuid: "49f8ec6ad17d4f818f3f88b7a864b1df" member_type: VOTER last_known_addr { host: "127.3.243.124" port: 43433 } } }
I20260619 18:52:34.275914 26652 sys_catalog.cc:458] T 00000000000000000000000000000000 P 716297d21b414fec897a3c165d669318 [sys.catalog]: This master's current role is: LEADER
I20260619 18:52:34.277649 26689 mvcc.cc:205] Tried to move back new op lower bound from 7298642551782510592 to 7298642551492476928. Current Snapshot: MvccSnapshot[applied={T|T < 7298642551782510592}]
I20260619 18:52:34.278378 26666 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49f8ec6ad17d4f818f3f88b7a864b1df [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "716297d21b414fec897a3c165d669318" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716297d21b414fec897a3c165d669318" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 43087 } } peers { permanent_uuid: "2e6bbc469c1c4f95bb2f58f5117bd358" member_type: VOTER last_known_addr { host: "127.3.243.125" port: 36419 } } peers { permanent_uuid: "49f8ec6ad17d4f818f3f88b7a864b1df" member_type: VOTER last_known_addr { host: "127.3.243.124" port: 43433 } } }
I20260619 18:52:34.278621 26666 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49f8ec6ad17d4f818f3f88b7a864b1df [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:52:34.278823 26571 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.243.124" port: 43433 } attempt: 5
I20260619 18:52:34.282995 26684 catalog_manager.cc:1347] Generated new cluster ID: ca161de9c1064afab3719f6ecd6e5a99
I20260619 18:52:34.283106 26684 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 18:52:34 dist-test-slave-m9wg krb5kdc[26417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895153, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.125@KRBTEST.COM for kudu/127.3.243.124@KRBTEST.COM
I20260619 18:52:34.295526 26574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:34.279364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.124:43433 (local address 127.0.0.1:34358)
0619 18:52:34.279672 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0619 18:52:34.279691 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 18:52:34.279797 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:52:34.280468 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:52:34.280473 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:52:34.281000 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:52:34.281204 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:52:34.281217 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:52:34.282357 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:52:34.282362 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:52:34.283937 (+ 1575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:52:34.283952 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:52:34.284873 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:52:34.284878 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:52:34.285088 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:34.285750 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:52:34.285773 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:52:34.289129 (+ 3356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:52:34.292664 (+ 3535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:52:34.292684 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:52:34.292689 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:52:34.293192 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:52:34.293688 (+ 496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:52:34.293707 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:52:34.293711 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:52:34.293881 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:52:34.294555 (+ 674us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:52:34.294560 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:52:34.294715 (+ 155us) client_negotiation.cc:769] Sending connection context
0619 18:52:34.294855 (+ 140us) client_negotiation.cc:240] Negotiation successful
0619 18:52:34.295133 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193}
I20260619 18:52:34.296088 26648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:34.279987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34358 (local address 127.3.243.124:43433)
0619 18:52:34.280163 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:34.280168 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:34.280186 (+ 18us) server_negotiation.cc:407] Connection header received
0619 18:52:34.280233 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:34.280237 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:34.280307 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:34.280399 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:34.281350 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:34.282214 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:34.284081 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:34.284738 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:34.284849 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:34.289301 (+ 4452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:52:34.289307 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:52:34.289310 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:52:34.289339 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:52:34.292541 (+ 3202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:34.293332 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:34.293340 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:34.293344 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:34.293550 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:34.294002 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:34.294007 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:34.294011 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:34.294345 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:52:34.294485 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:34.295077 (+ 592us) server_negotiation.cc:299] Negotiation successful
0619 18:52:34.295474 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 18:52:34.301131 26571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e6bbc469c1c4f95bb2f58f5117bd358: Bootstrap starting.
I20260619 18:52:34.304780 26571 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e6bbc469c1c4f95bb2f58f5117bd358: Neither blocks nor log segments found. Creating new log.
I20260619 18:52:34.306277 26571 log.cc:824] T 00000000000000000000000000000000 P 2e6bbc469c1c4f95bb2f58f5117bd358: Log is configured to *not* fsync() on all Append() calls
I20260619 18:52:34.309585 26571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e6bbc469c1c4f95bb2f58f5117bd358: No bootstrap required, opened a new log
I20260619 18:52:34.315667 26684 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:52:34.317480 26684 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:52:34.320459 26571 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e6bbc469c1c4f95bb2f58f5117bd358 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716297d21b414fec897a3c165d669318" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 43087 } } peers { permanent_uuid: "2e6bbc469c1c4f95bb2f58f5117bd358" member_type: VOTER last_known_addr { host: "127.3.243.125" port: 36419 } } peers { permanent_uuid: "49f8ec6ad17d4f818f3f88b7a864b1df" member_type: VOTER last_known_addr { host: "127.3.243.124" port: 43433 } }
I20260619 18:52:34.320837 26571 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e6bbc469c1c4f95bb2f58f5117bd358 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e6bbc469c1c4f95bb2f58f5117bd358, State: Initialized, Role: FOLLOWER
I20260619 18:52:34.321348 26571 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e6bbc469c1c4f95bb2f58f5117bd358 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716297d21b414fec897a3c165d669318" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 43087 } } peers { permanent_uuid: "2e6bbc469c1c4f95bb2f58f5117bd358" member_type: VOTER last_known_addr { host: "127.3.243.125" port: 36419 } } peers { permanent_uuid: "49f8ec6ad17d4f818f3f88b7a864b1df" member_type: VOTER last_known_addr { host: "127.3.243.124" port: 43433 } }
I20260619 18:52:34.322706 26694 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e6bbc469c1c4f95bb2f58f5117bd358 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716297d21b414fec897a3c165d669318" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 43087 } } peers { permanent_uuid: "2e6bbc469c1c4f95bb2f58f5117bd358" member_type: VOTER last_known_addr { host: "127.3.243.125" port: 36419 } } peers { permanent_uuid: "49f8ec6ad17d4f818f3f88b7a864b1df" member_type: VOTER last_known_addr { host: "127.3.243.124" port: 43433 } } }
I20260619 18:52:34.323518 26694 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e6bbc469c1c4f95bb2f58f5117bd358 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:52:34.323837 26571 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e6bbc469c1c4f95bb2f58f5117bd358 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:52:34.324301 26546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "716297d21b414fec897a3c165d669318" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "2e6bbc469c1c4f95bb2f58f5117bd358"
I20260619 18:52:34.324889 26545 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e6bbc469c1c4f95bb2f58f5117bd358 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:52:34.328770 26545 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2e6bbc469c1c4f95bb2f58f5117bd358 [term 1 FOLLOWER]: Refusing update from remote peer 716297d21b414fec897a3c165d669318: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:52:34.329761 26652 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 716297d21b414fec897a3c165d669318 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e6bbc469c1c4f95bb2f58f5117bd358" member_type: VOTER last_known_addr { host: "127.3.243.125" port: 36419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:52:34.340862 26684 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 716297d21b414fec897a3c165d669318: Generated new TSK 0
I20260619 18:52:34.341840 26684 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:52:34.401324 26546 raft_consensus.cc:2396] T 00000000000000000000000000000000 P 2e6bbc469c1c4f95bb2f58f5117bd358 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 716297d21b414fec897a3c165d669318 for term 1 because replica has last-logged OpId of term: 1 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260619 18:52:34.401718 26694 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e6bbc469c1c4f95bb2f58f5117bd358 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 716297d21b414fec897a3c165d669318. Latest consensus state: current_term: 1 leader_uuid: "716297d21b414fec897a3c165d669318" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716297d21b414fec897a3c165d669318" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 43087 } } peers { permanent_uuid: "2e6bbc469c1c4f95bb2f58f5117bd358" member_type: VOTER last_known_addr { host: "127.3.243.125" port: 36419 } } peers { permanent_uuid: "49f8ec6ad17d4f818f3f88b7a864b1df" member_type: VOTER last_known_addr { host: "127.3.243.124" port: 43433 } } }
I20260619 18:52:34.402045 26694 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e6bbc469c1c4f95bb2f58f5117bd358 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:52:34.402976 26702 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e6bbc469c1c4f95bb2f58f5117bd358 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "716297d21b414fec897a3c165d669318" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716297d21b414fec897a3c165d669318" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 43087 } } peers { permanent_uuid: "2e6bbc469c1c4f95bb2f58f5117bd358" member_type: VOTER last_known_addr { host: "127.3.243.125" port: 36419 } } peers { permanent_uuid: "49f8ec6ad17d4f818f3f88b7a864b1df" member_type: VOTER last_known_addr { host: "127.3.243.124" port: 43433 } } }
I20260619 18:52:34.403355 26702 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e6bbc469c1c4f95bb2f58f5117bd358 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:52:34.403326 26671 sys_catalog.cc:455] T 00000000000000000000000000000000 P 716297d21b414fec897a3c165d669318 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "716297d21b414fec897a3c165d669318" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "716297d21b414fec897a3c165d669318" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 43087 } } peers { permanent_uuid: "2e6bbc469c1c4f95bb2f58f5117bd358" member_type: VOTER last_known_addr { host: "127.3.243.125" port: 36419 } } peers { permanent_uuid: "49f8ec6ad17d4f818f3f88b7a864b1df" member_type: VOTER last_known_addr { host: "127.3.243.124" port: 43433 } } }
I20260619 18:52:34.403635 26671 sys_catalog.cc:458] T 00000000000000000000000000000000 P 716297d21b414fec897a3c165d669318 [sys.catalog]: This master's current role is: LEADER
I20260619 18:52:34.411537 26711 catalog_manager.cc:1259] Loaded cluster ID: ca161de9c1064afab3719f6ecd6e5a99
I20260619 18:52:34.411755 26711 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2e6bbc469c1c4f95bb2f58f5117bd358: loading cluster ID for follower catalog manager: success
I20260619 18:52:34.416610 26711 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2e6bbc469c1c4f95bb2f58f5117bd358: acquiring CA information for follower catalog manager: success
I20260619 18:52:34.418975 26711 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2e6bbc469c1c4f95bb2f58f5117bd358: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260619 18:52:34.458097 26501 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:52:34.442440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55286 (local address 127.3.243.126:43087)
0619 18:52:34.442624 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:34.442630 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:34.455036 (+ 12406us) server_negotiation.cc:407] Connection header received
0619 18:52:34.456450 (+ 1414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:34.456471 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:34.456589 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:34.456670 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:34.457853 (+ 1183us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55286: BlockingRecv error: recv got EOF from 127.0.0.1:55286 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260619 18:52:34.461792 26575 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:52:34.442134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41842 (local address 127.3.243.125:36419)
0619 18:52:34.442363 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:34.442369 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:34.459701 (+ 17332us) server_negotiation.cc:407] Connection header received
0619 18:52:34.460412 (+ 711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:34.460418 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:34.460510 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:34.460582 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:34.461567 (+ 985us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41842: BlockingRecv error: recv got EOF from 127.0.0.1:41842 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260619 18:52:34.464955 26648 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:52:34.455652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34372 (local address 127.3.243.124:43433)
0619 18:52:34.455807 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:34.455813 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:34.462661 (+ 6848us) server_negotiation.cc:407] Connection header received
0619 18:52:34.463415 (+ 754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:34.463422 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:34.463517 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:34.463589 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:34.464708 (+ 1119us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34372: BlockingRecv error: recv got EOF from 127.0.0.1:34372 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 18:52:34.479125 26575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:34.471614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41850 (local address 127.3.243.125:36419)
0619 18:52:34.471795 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:34.471801 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:34.472286 (+ 485us) server_negotiation.cc:407] Connection header received
0619 18:52:34.472498 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:34.472504 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:34.472702 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:34.472872 (+ 170us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:52:34.473316 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:34.474446 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:34.476419 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:34.477159 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:34.477313 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:34.477833 (+ 520us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:52:34.478481 (+ 648us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:52:34.478619 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:34.478712 (+ 93us) server_negotiation.cc:299] Negotiation successful
0619 18:52:34.478803 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260619 18:52:34.492682 26648 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:52:34.487047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34374 (local address 127.3.243.124:43433)
0619 18:52:34.487223 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:34.487229 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:34.491173 (+ 3944us) server_negotiation.cc:407] Connection header received
0619 18:52:34.491382 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:34.491388 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:34.491518 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:34.491622 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:34.492475 (+ 853us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34374: BlockingRecv error: recv got EOF from 127.0.0.1:34374 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 18:52:34.506916 26575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:34.497687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41862 (local address 127.3.243.125:36419)
0619 18:52:34.497902 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:34.497908 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:34.500796 (+ 2888us) server_negotiation.cc:407] Connection header received
0619 18:52:34.500942 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:34.500966 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:34.501059 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:34.501159 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:52:34.501664 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:34.502799 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:34.504731 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:34.505438 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:34.505568 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:34.506069 (+ 501us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:52:34.506237 (+ 168us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:52:34.506300 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:34.506524 (+ 224us) server_negotiation.cc:299] Negotiation successful
0619 18:52:34.506607 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260619 18:52:34.516680 26648 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:52:34.514268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34390 (local address 127.3.243.124:43433)
0619 18:52:34.514458 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:34.514464 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:34.514938 (+ 474us) server_negotiation.cc:407] Connection header received
0619 18:52:34.515216 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:34.515222 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:34.515329 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:34.515412 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:34.516427 (+ 1015us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34390: BlockingRecv error: recv got EOF from 127.0.0.1:34390 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 18:52:34.534823 26575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:34.522413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41872 (local address 127.3.243.125:36419)
0619 18:52:34.522577 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:34.522583 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:34.528687 (+ 6104us) server_negotiation.cc:407] Connection header received
0619 18:52:34.528853 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:34.528859 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:34.528941 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:34.529032 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:52:34.529536 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:34.530708 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:34.532646 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:34.533315 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:34.533433 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:34.533922 (+ 489us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:52:34.534101 (+ 179us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:52:34.534187 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:34.534407 (+ 220us) server_negotiation.cc:299] Negotiation successful
0619 18:52:34.534500 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260619 18:52:34.546011 26648 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:52:34.541776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34392 (local address 127.3.243.124:43433)
0619 18:52:34.541961 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:34.541966 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:34.544640 (+ 2674us) server_negotiation.cc:407] Connection header received
0619 18:52:34.544812 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:34.544818 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:34.544908 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:34.545007 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:34.545795 (+ 788us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34392: BlockingRecv error: recv got EOF from 127.0.0.1:34392 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 18:52:34.560974 26575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:34.554337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41876 (local address 127.3.243.125:36419)
0619 18:52:34.554569 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:34.554574 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:34.555051 (+ 477us) server_negotiation.cc:407] Connection header received
0619 18:52:34.555216 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:34.555222 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:34.555321 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:34.555426 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:52:34.555858 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:34.557025 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:34.558886 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:34.559580 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:34.559696 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:34.560228 (+ 532us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:52:34.560356 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:52:34.560440 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:34.560601 (+ 161us) server_negotiation.cc:299] Negotiation successful
0619 18:52:34.560684 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
W20260619 18:52:34.602581 26648 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:52:34.589979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34398 (local address 127.3.243.124:43433)
0619 18:52:34.590237 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:34.590244 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:34.600839 (+ 10595us) server_negotiation.cc:407] Connection header received
0619 18:52:34.601067 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:34.601073 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:34.601174 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:34.601282 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:34.602345 (+ 1063us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34398: BlockingRecv error: recv got EOF from 127.0.0.1:34398 (error 108)
Metrics: {"server-negotiator.queue_time_us":138}
I20260619 18:52:34.620453 26575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:34.613899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41878 (local address 127.3.243.125:36419)
0619 18:52:34.614078 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:34.614084 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:34.614556 (+ 472us) server_negotiation.cc:407] Connection header received
0619 18:52:34.614720 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:34.614727 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:34.614817 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:34.614916 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:52:34.615341 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:34.616531 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:34.618401 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:34.619084 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:34.619210 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:34.619664 (+ 454us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:52:34.619798 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:52:34.619885 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:34.620030 (+ 145us) server_negotiation.cc:299] Negotiation successful
0619 18:52:34.620116 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260619 18:52:34.636456 26648 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:52:34.627033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34410 (local address 127.3.243.124:43433)
0619 18:52:34.627203 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:34.627209 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:34.632704 (+ 5495us) server_negotiation.cc:407] Connection header received
0619 18:52:34.634477 (+ 1773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:34.634484 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:34.634623 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:34.634717 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:34.636062 (+ 1345us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34410: BlockingRecv error: recv got EOF from 127.0.0.1:34410 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 18:52:34.663306 26575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:34.657719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41892 (local address 127.3.243.125:36419)
0619 18:52:34.657901 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:34.657907 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:34.658174 (+ 267us) server_negotiation.cc:407] Connection header received
0619 18:52:34.658334 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:34.658339 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:34.658424 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:34.658535 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:52:34.658860 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:34.659988 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:34.661301 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:34.661938 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:34.662067 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:34.662526 (+ 459us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:52:34.662706 (+ 180us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:52:34.662798 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:34.662903 (+ 105us) server_negotiation.cc:299] Negotiation successful
0619 18:52:34.662999 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260619 18:52:34.670563 26648 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:52:34.668749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34414 (local address 127.3.243.124:43433)
0619 18:52:34.668932 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:34.668937 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:34.669238 (+ 301us) server_negotiation.cc:407] Connection header received
0619 18:52:34.669423 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:34.669427 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:34.669494 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:34.669570 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:34.670342 (+ 772us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34414: BlockingRecv error: recv got EOF from 127.0.0.1:34414 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 18:52:34.712327 26575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:34.706039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41900 (local address 127.3.243.125:36419)
0619 18:52:34.706273 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:34.706279 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:34.707000 (+ 721us) server_negotiation.cc:407] Connection header received
0619 18:52:34.707166 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:34.707172 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:34.707282 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:34.707402 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:52:34.707787 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:34.708987 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:34.710319 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:34.711064 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:34.711187 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:34.711549 (+ 362us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:52:34.711696 (+ 147us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:52:34.711791 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:34.711894 (+ 103us) server_negotiation.cc:299] Negotiation successful
0619 18:52:34.711972 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260619 18:52:34.719906 26648 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:52:34.718018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34426 (local address 127.3.243.124:43433)
0619 18:52:34.718214 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:34.718220 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:34.718476 (+ 256us) server_negotiation.cc:407] Connection header received
0619 18:52:34.718664 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:34.718668 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:34.718737 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:34.718825 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:34.719696 (+ 871us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34426: BlockingRecv error: recv got EOF from 127.0.0.1:34426 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 18:52:34.793561 26575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:34.787849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41902 (local address 127.3.243.125:36419)
0619 18:52:34.788043 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:34.788050 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:34.788369 (+ 319us) server_negotiation.cc:407] Connection header received
0619 18:52:34.788558 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:34.788564 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:34.788656 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:34.788792 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:52:34.789123 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:34.790274 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:34.791601 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:34.792317 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:34.792465 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:34.792857 (+ 392us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:52:34.793003 (+ 146us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:52:34.793090 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:34.793184 (+ 94us) server_negotiation.cc:299] Negotiation successful
0619 18:52:34.793265 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260619 18:52:34.801574 26648 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:52:34.799654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34434 (local address 127.3.243.124:43433)
0619 18:52:34.799844 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:34.799850 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:34.800210 (+ 360us) server_negotiation.cc:407] Connection header received
0619 18:52:34.800419 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:34.800425 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:34.800542 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:34.800645 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:34.801320 (+ 675us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34434: BlockingRecv error: recv got EOF from 127.0.0.1:34434 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 18:52:34.937348 26575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:34.932669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41906 (local address 127.3.243.125:36419)
0619 18:52:34.932888 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:34.932895 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:34.933113 (+ 218us) server_negotiation.cc:407] Connection header received
0619 18:52:34.933268 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:34.933272 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:34.933339 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:34.933414 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:52:34.933827 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:34.934513 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:34.935841 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:34.936305 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:34.936399 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:34.936736 (+ 337us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:52:34.936872 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:52:34.936933 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:34.937039 (+ 106us) server_negotiation.cc:299] Negotiation successful
0619 18:52:34.937098 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260619 18:52:34.944193 26648 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:52:34.942470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34436 (local address 127.3.243.124:43433)
0619 18:52:34.942692 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:34.942698 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:34.942923 (+ 225us) server_negotiation.cc:407] Connection header received
0619 18:52:34.943102 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:34.943108 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:34.943188 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:34.943278 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:34.943953 (+ 675us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34436: BlockingRecv error: recv got EOF from 127.0.0.1:34436 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 18:52:35.208022 26575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:35.203434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41916 (local address 127.3.243.125:36419)
0619 18:52:35.203640 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:35.203646 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:35.203749 (+ 103us) server_negotiation.cc:407] Connection header received
0619 18:52:35.203970 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:35.203975 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:35.204042 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:35.204116 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:52:35.204565 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:35.205257 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:35.206512 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:35.206919 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:35.207023 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:35.207395 (+ 372us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:52:35.207494 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:52:35.207563 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:35.207696 (+ 133us) server_negotiation.cc:299] Negotiation successful
0619 18:52:35.207780 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260619 18:52:35.215008 26648 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:52:35.213410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34452 (local address 127.3.243.124:43433)
0619 18:52:35.213637 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:35.213641 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:35.213809 (+ 168us) server_negotiation.cc:407] Connection header received
0619 18:52:35.213935 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:35.213939 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:35.214007 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:35.214079 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:35.214743 (+ 664us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34452: BlockingRecv error: recv got EOF from 127.0.0.1:34452 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 18:52:35.255690 26688 catalog_manager.cc:1259] Loaded cluster ID: ca161de9c1064afab3719f6ecd6e5a99
I20260619 18:52:35.255806 26688 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 49f8ec6ad17d4f818f3f88b7a864b1df: loading cluster ID for follower catalog manager: success
I20260619 18:52:35.258404 26688 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 49f8ec6ad17d4f818f3f88b7a864b1df: acquiring CA information for follower catalog manager: success
I20260619 18:52:35.260104 26688 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 49f8ec6ad17d4f818f3f88b7a864b1df: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 18:52:35.734838 26827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:35.730101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41928 (local address 127.3.243.125:36419)
0619 18:52:35.730812 (+ 711us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:35.730816 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:35.730833 (+ 17us) server_negotiation.cc:407] Connection header received
0619 18:52:35.730896 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:35.730899 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:35.730979 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:35.731062 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:52:35.731452 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:35.732210 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:35.733479 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:35.733853 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:35.733938 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:35.734224 (+ 286us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:52:35.734319 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:52:35.734377 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:35.734484 (+ 107us) server_negotiation.cc:299] Negotiation successful
0619 18:52:35.734544 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":521,"threads_started":1}
I20260619 18:52:35.745594 26833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:35.740336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34460 (local address 127.3.243.124:43433)
0619 18:52:35.740939 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:35.740943 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:35.740959 (+ 16us) server_negotiation.cc:407] Connection header received
0619 18:52:35.741012 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:35.741016 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:35.741155 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:35.741258 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:52:35.741592 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:35.742326 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:35.743511 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:35.743927 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:35.744030 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:35.744580 (+ 550us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:52:35.745099 (+ 519us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:52:35.745211 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:35.745311 (+ 100us) server_negotiation.cc:299] Negotiation successful
0619 18:52:35.745386 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":423,"threads_started":1}
I20260619 18:52:35.758385 26827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:35.751216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41940 (local address 127.3.243.125:36419)
0619 18:52:35.751393 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:35.751400 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:35.752357 (+ 957us) server_negotiation.cc:407] Connection header received
0619 18:52:35.752631 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:35.752637 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:35.752704 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:35.752763 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:52:35.753561 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:35.754553 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:35.756611 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:35.757182 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:35.757312 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:35.757698 (+ 386us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:52:35.757809 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:52:35.757893 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:35.758024 (+ 131us) server_negotiation.cc:299] Negotiation successful
0619 18:52:35.758100 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 18:52:35.758512 26833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:35.752334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34468 (local address 127.3.243.124:43433)
0619 18:52:35.752489 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:35.752494 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:35.752749 (+ 255us) server_negotiation.cc:407] Connection header received
0619 18:52:35.752935 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:35.752939 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:35.753021 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:35.753070 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:52:35.753862 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:35.754825 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:35.756449 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:35.756919 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:35.757023 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:35.757824 (+ 801us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:52:35.757932 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:52:35.758000 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:35.758183 (+ 183us) server_negotiation.cc:299] Negotiation successful
0619 18:52:35.758253 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 18:52:35.759259 26840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:35.751217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55290 (local address 127.3.243.126:43087)
0619 18:52:35.751721 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:35.751728 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:35.752118 (+ 390us) server_negotiation.cc:407] Connection header received
0619 18:52:35.752456 (+ 338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:35.752463 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:35.752694 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:35.752866 (+ 172us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:52:35.753475 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:35.754303 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:35.756359 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:35.756906 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:35.757024 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:35.757499 (+ 475us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:52:35.757922 (+ 423us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:52:35.758037 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:35.758913 (+ 876us) server_negotiation.cc:299] Negotiation successful
0619 18:52:35.758992 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":266,"threads_started":1}
I20260619 18:52:35.763563 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 26430
I20260619 18:52:35.764626 26492 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:52:35.876996 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 26430
I20260619 18:52:35.893846 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 26504
I20260619 18:52:35.894941 26566 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:52:36.000453 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 26504
I20260619 18:52:36.017890 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 26578
I20260619 18:52:36.018899 26640 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:52:36.124358 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 26578
2026-06-19T18:52:36Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4719 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894810715282-4045-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:52:36 dist-test-slave-m9wg krb5kdc[26847](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:52:36 dist-test-slave-m9wg krb5kdc[26847](info): set up 2 sockets
Jun 19 18:52:36 dist-test-slave-m9wg krb5kdc[26847](info): commencing operation
krb5kdc: starting...
W20260619 18:52:37.132545 4045 mini_kdc.cc:121] Time spent starting KDC: real 0.972s user 0.001s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894810715282-4045-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894810715282-4045-0/krb5kdc/test-user.keytab.
Jun 19 18:52:37 dist-test-slave-m9wg krb5kdc[26847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895157, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:52:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:52:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.243.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab.
Entry for principal kudu/127.3.243.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab.
I20260619 18:52:37.357131 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894810715282-4045-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894810715282-4045-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894810715282-4045-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.126:38079
--webserver_interface=127.3.243.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab
--principal=kudu/127.3.243.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:44601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.126:38079,127.3.243.125:37293,127.3.243.124:33211
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:52:37.668263 26860 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 18:52:37.668579 26860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:52:37.668673 26860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:52:37.668787 26860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:52:37.675748 26860 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:52:37.675829 26860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:52:37.675913 26860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:52:37.675976 26860 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:52:37.676054 26860 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:52:37.685424 26860 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:44601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894810715282-4045-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894810715282-4045-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.243.126:38079,127.3.243.125:37293,127.3.243.124:33211
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.126.keytab
--principal=kudu/127.3.243.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.126:38079
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.243.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894810715282-4045-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:52:37.686570 26860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:52:37.687849 26860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:52:37.698367 26865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:52:37.698415 26868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:52:37.698446 26866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:52:37.699277 26860 server_base.cc:1034] running on GCE node
I20260619 18:52:37.699924 26860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:52:37.701139 26860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:52:37.702342 26860 hybrid_clock.cc:648] HybridClock initialized: now 1781895157702296 us; error 39 us; skew 500 ppm
I20260619 18:52:37.702606 26860 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 18:52:37 dist-test-slave-m9wg krb5kdc[26847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895157, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:52:37.711808 26860 init.cc:377] Logged in from keytab as kudu/127.3.243.126@KRBTEST.COM (short username kudu)
I20260619 18:52:37.713150 26860 webserver.cc:460] Webserver started at http://127.3.243.126:34381/ using document root <none> and password file <none>
I20260619 18:52:37.713786 26860 fs_manager.cc:359] Metadata directory not provided
I20260619 18:52:37.713881 26860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:52:37.714159 26860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:52:37.716940 26860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894810715282-4045-0/minicluster-data/master-0/data/instance:
uuid: "020a26e9047d461fb3be0ed825316f89"
format_stamp: "Formatted at 2026-06-19 18:52:37 on dist-test-slave-m9wg"
server_key: "dc7bd8e020ecc1dbec89921d0ebc87c2"
server_key_iv: "1d17638eff9224633d1b08c842b82480"
server_key_version: "encryptionkey@0"
I20260619 18:52:37.717650 26860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894810715282-4045-0/minicluster-data/master-0/wal/instance:
uuid: "020a26e9047d461fb3be0ed825316f89"
format_stamp: "Formatted at 2026-06-19 18:52:37 on dist-test-slave-m9wg"
server_key: "dc7bd8e020ecc1dbec89921d0ebc87c2"
server_key_iv: "1d17638eff9224633d1b08c842b82480"
server_key_version: "encryptionkey@0"
I20260619 18:52:37.722311 26860 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 18:52:37.725449 26875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:52:37.726625 26860 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 18:52:37.726799 26860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894810715282-4045-0/minicluster-data/master-0/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894810715282-4045-0/minicluster-data/master-0/wal
uuid: "020a26e9047d461fb3be0ed825316f89"
format_stamp: "Formatted at 2026-06-19 18:52:37 on dist-test-slave-m9wg"
server_key: "dc7bd8e020ecc1dbec89921d0ebc87c2"
server_key_iv: "1d17638eff9224633d1b08c842b82480"
server_key_version: "encryptionkey@0"
I20260619 18:52:37.726980 26860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894810715282-4045-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894810715282-4045-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894810715282-4045-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:52:37.744462 26860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:52:37.748303 26860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:52:37.748538 26860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:52:37.765395 26860 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.126:38079
I20260619 18:52:37.765414 26926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.126:38079 every 8 connection(s)
I20260619 18:52:37.766656 26860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894810715282-4045-0/minicluster-data/master-0/data/info.pb
I20260619 18:52:37.769019 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 26860
I20260619 18:52:37.769200 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894810715282-4045-0/minicluster-data/master-0/wal/instance
I20260619 18:52:37.769721 4045 external_mini_cluster.cc:1428] Setting key f651f2ca0ac6ebf1c6a3b8372496ade8
I20260619 18:52:37.770740 26927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:52:37.778934 26927 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.243.126" port: 38079 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 18:52:37 dist-test-slave-m9wg krb5kdc[26847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895157, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.126@KRBTEST.COM for kudu/127.3.243.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.243.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.125@KRBTEST.COM" created.
I20260619 18:52:37.800829 26931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:37.783773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42526 (local address 127.3.243.126:38079)
0619 18:52:37.784447 (+ 674us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:37.784455 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:37.784511 (+ 56us) server_negotiation.cc:407] Connection header received
0619 18:52:37.785650 (+ 1139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:37.785655 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:37.785891 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:37.786061 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:37.787290 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:37.788524 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:37.789842 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:37.790378 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:37.790546 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:37.793884 (+ 3338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:52:37.793892 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:52:37.793903 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:52:37.793931 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:52:37.796606 (+ 2675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:37.797245 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:37.797259 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:37.797261 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:37.797415 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:37.797851 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:37.797855 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:37.797858 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:37.798362 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:52:37.798553 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:37.799056 (+ 503us) server_negotiation.cc:299] Negotiation successful
0619 18:52:37.799369 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":7,"server-negotiator.queue_time_us":458,"thread_start_us":287,"threads_started":1}
I20260619 18:52:37.800800 26930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:37.782563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:38079 (local address 127.0.0.1:42526)
0619 18:52:37.783287 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0619 18:52:37.783331 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 18:52:37.785315 (+ 1984us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:52:37.786101 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:52:37.786108 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:52:37.786756 (+ 648us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:52:37.787091 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:52:37.787108 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:52:37.788662 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:52:37.788666 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:52:37.789715 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:52:37.789727 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:52:37.790596 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:52:37.790603 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:52:37.790843 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:37.791467 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:52:37.791488 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:52:37.793672 (+ 2184us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:52:37.796722 (+ 3050us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:52:37.796732 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:52:37.796745 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:52:37.797128 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:52:37.797516 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:52:37.797521 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:52:37.797526 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:52:37.797714 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:52:37.798501 (+ 787us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:52:37.798509 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:52:37.798695 (+ 186us) client_negotiation.cc:769] Sending connection context
0619 18:52:37.798924 (+ 229us) client_negotiation.cc:240] Negotiation successful
0619 18:52:37.799501 (+ 577us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"mutex_wait_us":197,"thread_start_us":284,"threads_started":1}
I20260619 18:52:37.804097 26927 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.243.125" port: 37293 } has no permanent_uuid. Determining permanent_uuid...
I20260619 18:52:37.805204 26930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:37.804792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.125:37293 (local address 127.0.0.1:48702)
0619 18:52:37.804959 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0619 18:52:37.805075 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.125:37293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260619 18:52:37.805531 26878 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.243.125:37293: connect: Connection refused (error 111)
I20260619 18:52:37.807121 26930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:37.806706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.125:37293 (local address 127.0.0.1:48714)
0619 18:52:37.806917 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0619 18:52:37.806996 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.125:37293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
W20260619 18:52:37.807333 26927 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.243.125:37293: Network error: Client connection negotiation failed: client connection to 127.3.243.125:37293: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.125.keytab.
Entry for principal kudu/127.3.243.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.125.keytab.
I20260619 18:52:37.825459 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894810715282-4045-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894810715282-4045-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894810715282-4045-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894810715282-4045-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.125:37293
--webserver_interface=127.3.243.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.125.keytab
--principal=kudu/127.3.243.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:44601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.126:38079,127.3.243.125:37293,127.3.243.124:33211
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 18:52:37.856643 26927 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.243.125" port: 37293 } attempt: 1
I20260619 18:52:37.857843 26930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:37.857390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.125:37293 (local address 127.0.0.1:48728)
0619 18:52:37.857585 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0619 18:52:37.857698 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.125:37293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260619 18:52:37.858779 26930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:37.858470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.125:37293 (local address 127.0.0.1:48730)
0619 18:52:37.858590 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0619 18:52:37.858656 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.125:37293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260619 18:52:37.858945 26927 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.243.125:37293: Network error: Client connection negotiation failed: client connection to 127.3.243.125:37293: connect: Connection refused (error 111)
I20260619 18:52:37.927167 26927 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.243.125" port: 37293 } attempt: 2
I20260619 18:52:37.928367 26930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:37.927858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.125:37293 (local address 127.0.0.1:48732)
0619 18:52:37.928084 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0619 18:52:37.928212 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.125:37293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260619 18:52:37.929261 26930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:37.928959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.125:37293 (local address 127.0.0.1:48740)
0619 18:52:37.929070 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0619 18:52:37.929147 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.125:37293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260619 18:52:37.929454 26927 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.243.125:37293: Network error: Client connection negotiation failed: client connection to 127.3.243.125:37293: connect: Connection refused (error 111)
I20260619 18:52:38.020678 26927 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.243.125" port: 37293 } attempt: 3
I20260619 18:52:38.021801 26930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:38.021396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.125:37293 (local address 127.0.0.1:48754)
0619 18:52:38.021580 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0619 18:52:38.021670 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.125:37293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260619 18:52:38.022719 26930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:38.022403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.125:37293 (local address 127.0.0.1:48756)
0619 18:52:38.022524 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0619 18:52:38.022591 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.125:37293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260619 18:52:38.022922 26927 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.243.125:37293: Network error: Client connection negotiation failed: client connection to 127.3.243.125:37293: connect: Connection refused (error 111)
W20260619 18:52:38.128189 26934 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 18:52:38.128471 26934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:52:38.128539 26934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:52:38.128616 26934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:52:38.135600 26934 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:52:38.135679 26934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:52:38.135746 26934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:52:38.135798 26934 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:52:38.135859 26934 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:52:38.145119 26934 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:44601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894810715282-4045-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894810715282-4045-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.243.126:38079,127.3.243.125:37293,127.3.243.124:33211
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.125.keytab
--principal=kudu/127.3.243.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.125:37293
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894810715282-4045-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.243.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894810715282-4045-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:52:38.146310 26934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:52:38.147495 26934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:52:38.158550 26942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:52:38.158593 26939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:52:38.158510 26940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:52:38.161412 26934 server_base.cc:1034] running on GCE node
I20260619 18:52:38.162130 26934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:52:38.163456 26934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:52:38.164680 26934 hybrid_clock.cc:648] HybridClock initialized: now 1781895158164608 us; error 43 us; skew 500 ppm
I20260619 18:52:38.165020 26934 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 18:52:38.166115 26927 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.243.125" port: 37293 } attempt: 4
I20260619 18:52:38.167098 26930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:38.166616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.125:37293 (local address 127.0.0.1:48762)
0619 18:52:38.166892 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0619 18:52:38.166972 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.125:37293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128}
Jun 19 18:52:38 dist-test-slave-m9wg krb5kdc[26847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895158, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:52:38.168210 26934 init.cc:377] Logged in from keytab as kudu/127.3.243.125@KRBTEST.COM (short username kudu)
I20260619 18:52:38.168550 26930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:38.168179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.125:37293 (local address 127.0.0.1:48772)
0619 18:52:38.168319 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 18:52:38.168405 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.125:37293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260619 18:52:38.168793 26927 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.243.125:37293: Network error: Client connection negotiation failed: client connection to 127.3.243.125:37293: connect: Connection refused (error 111)
I20260619 18:52:38.169706 26934 webserver.cc:460] Webserver started at http://127.3.243.125:43841/ using document root <none> and password file <none>
I20260619 18:52:38.170341 26934 fs_manager.cc:359] Metadata directory not provided
I20260619 18:52:38.170419 26934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:52:38.170663 26934 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:52:38.173460 26934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894810715282-4045-0/minicluster-data/master-1/data/instance:
uuid: "2b525a8d5f2247a689df2fbf1d4dcca7"
format_stamp: "Formatted at 2026-06-19 18:52:38 on dist-test-slave-m9wg"
server_key: "548447e5256db23e6b7858c7138892e0"
server_key_iv: "c98a73ccca6470e27e4e29158faae915"
server_key_version: "encryptionkey@0"
I20260619 18:52:38.174171 26934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894810715282-4045-0/minicluster-data/master-1/wal/instance:
uuid: "2b525a8d5f2247a689df2fbf1d4dcca7"
format_stamp: "Formatted at 2026-06-19 18:52:38 on dist-test-slave-m9wg"
server_key: "548447e5256db23e6b7858c7138892e0"
server_key_iv: "c98a73ccca6470e27e4e29158faae915"
server_key_version: "encryptionkey@0"
I20260619 18:52:38.179013 26934 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 18:52:38.182173 26949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:52:38.183470 26934 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 18:52:38.183619 26934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894810715282-4045-0/minicluster-data/master-1/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894810715282-4045-0/minicluster-data/master-1/wal
uuid: "2b525a8d5f2247a689df2fbf1d4dcca7"
format_stamp: "Formatted at 2026-06-19 18:52:38 on dist-test-slave-m9wg"
server_key: "548447e5256db23e6b7858c7138892e0"
server_key_iv: "c98a73ccca6470e27e4e29158faae915"
server_key_version: "encryptionkey@0"
I20260619 18:52:38.183802 26934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894810715282-4045-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894810715282-4045-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894810715282-4045-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:52:38.197702 26934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:52:38.201529 26934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:52:38.201766 26934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:52:38.217849 26934 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.125:37293
I20260619 18:52:38.217869 27000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.125:37293 every 8 connection(s)
I20260619 18:52:38.219070 26934 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894810715282-4045-0/minicluster-data/master-1/data/info.pb
I20260619 18:52:38.222465 27001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:52:38.227598 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 26934
I20260619 18:52:38.227785 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894810715282-4045-0/minicluster-data/master-1/wal/instance
I20260619 18:52:38.228305 4045 external_mini_cluster.cc:1428] Setting key 7eae6dcf0f479814415272ed39a2b8ca
I20260619 18:52:38.229352 27001 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.243.126" port: 38079 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 18:52:38 dist-test-slave-m9wg krb5kdc[26847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895158, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.125@KRBTEST.COM for kudu/127.3.243.126@KRBTEST.COM
I20260619 18:52:38.250038 26931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:38.232229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42528 (local address 127.3.243.126:38079)
0619 18:52:38.232430 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:38.232436 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:38.233567 (+ 1131us) server_negotiation.cc:407] Connection header received
0619 18:52:38.235810 (+ 2243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:38.235815 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:38.235895 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:38.235979 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:38.237729 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:38.238415 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:38.239893 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:38.240515 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:38.240636 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:38.243727 (+ 3091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:52:38.243733 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:52:38.243738 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:52:38.243774 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:52:38.247111 (+ 3337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:38.247888 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:38.247892 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:38.247895 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:38.248017 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:38.248455 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:38.248458 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:38.248461 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:38.248752 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:52:38.248863 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:38.249613 (+ 750us) server_negotiation.cc:299] Negotiation successful
0619 18:52:38.249815 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 18:52:38.251236 27003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:38.232662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:38079 (local address 127.0.0.1:42528)
0619 18:52:38.233396 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0619 18:52:38.233446 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 18:52:38.235515 (+ 2069us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:52:38.236246 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:52:38.236257 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:52:38.236902 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:52:38.237568 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:52:38.237589 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:52:38.238563 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:52:38.238568 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:52:38.239745 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:52:38.239757 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:52:38.240702 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:52:38.240707 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:52:38.240911 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:38.241510 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:52:38.241535 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:52:38.243516 (+ 1981us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:52:38.247267 (+ 3751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:52:38.247275 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:52:38.247288 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:52:38.247779 (+ 491us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:52:38.248111 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:52:38.248115 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:52:38.248119 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:52:38.248348 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:52:38.248858 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:52:38.248864 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:52:38.249236 (+ 372us) client_negotiation.cc:769] Sending connection context
0619 18:52:38.249512 (+ 276us) client_negotiation.cc:240] Negotiation successful
0619 18:52:38.249964 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":266,"threads_started":1}
I20260619 18:52:38.252919 27001 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.243.125" port: 37293 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.3.243.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.124@KRBTEST.COM" created.
Jun 19 18:52:38 dist-test-slave-m9wg krb5kdc[26847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895158, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.125@KRBTEST.COM for kudu/127.3.243.125@KRBTEST.COM
I20260619 18:52:38.267082 27003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:38.253748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.125:37293 (local address 127.0.0.1:48788)
0619 18:52:38.253938 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0619 18:52:38.253955 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 18:52:38.254028 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:52:38.254868 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:52:38.254873 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:52:38.255279 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:52:38.255400 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:52:38.255408 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:52:38.256977 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:52:38.256983 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:52:38.257886 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:52:38.257898 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:52:38.258667 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:52:38.258673 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:52:38.258827 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:38.259348 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:52:38.259366 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:52:38.261566 (+ 2200us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:52:38.264797 (+ 3231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:52:38.264802 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:52:38.264806 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:52:38.265279 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:52:38.265628 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:52:38.265631 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:52:38.265633 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:52:38.265802 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:52:38.266430 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:52:38.266434 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:52:38.266541 (+ 107us) client_negotiation.cc:769] Sending connection context
0619 18:52:38.266640 (+ 99us) client_negotiation.cc:240] Negotiation successful
0619 18:52:38.266849 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260619 18:52:38.267401 27005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:38.253879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48788 (local address 127.3.243.125:37293)
0619 18:52:38.254465 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:38.254471 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:38.254494 (+ 23us) server_negotiation.cc:407] Connection header received
0619 18:52:38.254530 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:38.254536 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:38.254740 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:38.254869 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:38.255561 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:38.256805 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:38.258012 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:38.258519 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:38.258625 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:38.261756 (+ 3131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:52:38.261767 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:52:38.261781 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:52:38.261816 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:52:38.264671 (+ 2855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:38.265401 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:38.265408 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:38.265411 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:38.265536 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:38.265905 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:38.265910 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:38.265913 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:38.266316 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:52:38.266492 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:38.266793 (+ 301us) server_negotiation.cc:299] Negotiation successful
0619 18:52:38.267016 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":355,"threads_started":1}
I20260619 18:52:38.269464 27001 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.243.124" port: 33211 } has no permanent_uuid. Determining permanent_uuid...
I20260619 18:52:38.270481 27003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:38.270092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.124:33211 (local address 127.0.0.1:53770)
0619 18:52:38.270257 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0619 18:52:38.270358 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.124:33211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260619 18:52:38.270707 26951 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.243.124:33211: connect: Connection refused (error 111)
I20260619 18:52:38.272101 27003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:38.271812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.124:33211 (local address 127.0.0.1:53778)
0619 18:52:38.271937 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0619 18:52:38.272007 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.124:33211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260619 18:52:38.272383 27001 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.243.124:33211: Network error: Client connection negotiation failed: client connection to 127.3.243.124:33211: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.124.keytab.
Entry for principal kudu/127.3.243.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.124.keytab.
I20260619 18:52:38.285409 4045 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
/tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894810715282-4045-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894810715282-4045-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894810715282-4045-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894810715282-4045-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.124:33211
--webserver_interface=127.3.243.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.124.keytab
--principal=kudu/127.3.243.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.84:44601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.126:38079,127.3.243.125:37293,127.3.243.124:33211
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894810715282-4045-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894810715282-4045-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894810715282-4045-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 18:52:38.321650 27001 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.243.124" port: 33211 } attempt: 1
I20260619 18:52:38.322674 27003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:38.322252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.124:33211 (local address 127.0.0.1:53792)
0619 18:52:38.322432 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0619 18:52:38.322532 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.124:33211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260619 18:52:38.323561 27003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:38.323241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.124:33211 (local address 127.0.0.1:53796)
0619 18:52:38.323387 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0619 18:52:38.323459 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.124:33211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260619 18:52:38.323743 27001 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.243.124:33211: Network error: Client connection negotiation failed: client connection to 127.3.243.124:33211: connect: Connection refused (error 111)
I20260619 18:52:38.391956 27001 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.243.124" port: 33211 } attempt: 2
I20260619 18:52:38.393059 27003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:38.392604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.124:33211 (local address 127.0.0.1:53800)
0619 18:52:38.392797 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0619 18:52:38.392905 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.124:33211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260619 18:52:38.393983 27003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:38.393639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.124:33211 (local address 127.0.0.1:53816)
0619 18:52:38.393790 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0619 18:52:38.393864 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.124:33211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260619 18:52:38.394182 27001 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.243.124:33211: Network error: Client connection negotiation failed: client connection to 127.3.243.124:33211: connect: Connection refused (error 111)
I20260619 18:52:38.468039 26927 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.243.125" port: 37293 } attempt: 5
Jun 19 18:52:38 dist-test-slave-m9wg krb5kdc[26847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895157, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.126@KRBTEST.COM for kudu/127.3.243.125@KRBTEST.COM
I20260619 18:52:38.480634 27005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:38.468875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48804 (local address 127.3.243.125:37293)
0619 18:52:38.469061 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:38.469066 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:38.469136 (+ 70us) server_negotiation.cc:407] Connection header received
0619 18:52:38.469331 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:38.469335 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:38.469417 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:38.469500 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:38.470320 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:38.471014 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:38.472191 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:38.472607 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:38.472725 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:38.475590 (+ 2865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:52:38.475595 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:52:38.475598 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:52:38.475630 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:52:38.478201 (+ 2571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:38.478840 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:38.478846 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:38.478849 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:38.478964 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:38.479305 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:38.479310 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:38.479314 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:38.479580 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:52:38.479709 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:38.480083 (+ 374us) server_negotiation.cc:299] Negotiation successful
0619 18:52:38.480322 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260619 18:52:38.480639 26930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:38.468815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.125:37293 (local address 127.0.0.1:48804)
0619 18:52:38.469033 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0619 18:52:38.469054 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 18:52:38.469199 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:52:38.469552 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:52:38.469557 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:52:38.469959 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:52:38.470151 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:52:38.470160 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:52:38.471150 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:52:38.471157 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:52:38.472016 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:52:38.472029 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:52:38.472757 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:52:38.472764 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:52:38.472952 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:38.473502 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:52:38.473519 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:52:38.475414 (+ 1895us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:52:38.478349 (+ 2935us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:52:38.478356 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:52:38.478360 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:52:38.478723 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:52:38.479071 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:52:38.479075 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:52:38.479079 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:52:38.479206 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:52:38.479691 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:52:38.479695 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:52:38.479853 (+ 158us) client_negotiation.cc:769] Sending connection context
0619 18:52:38.479970 (+ 117us) client_negotiation.cc:240] Negotiation successful
0619 18:52:38.480275 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260619 18:52:38.481788 26927 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.243.124" port: 33211 } has no permanent_uuid. Determining permanent_uuid...
I20260619 18:52:38.482831 26930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:38.482464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.124:33211 (local address 127.0.0.1:53832)
0619 18:52:38.482624 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 18:52:38.482697 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.124:33211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260619 18:52:38.483701 26930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:38.483405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.124:33211 (local address 127.0.0.1:53848)
0619 18:52:38.483535 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0619 18:52:38.483609 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.124:33211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260619 18:52:38.483886 26927 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.243.124:33211: Network error: Client connection negotiation failed: client connection to 127.3.243.124:33211: connect: Connection refused (error 111)
I20260619 18:52:38.485376 27001 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.243.124" port: 33211 } attempt: 3
I20260619 18:52:38.486188 27003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:38.485818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.124:33211 (local address 127.0.0.1:53854)
0619 18:52:38.485991 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0619 18:52:38.486064 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.124:33211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260619 18:52:38.487064 27003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:38.486680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.124:33211 (local address 127.0.0.1:53870)
0619 18:52:38.486890 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0619 18:52:38.486956 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.124:33211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
W20260619 18:52:38.487278 27001 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.243.124:33211: Network error: Client connection negotiation failed: client connection to 127.3.243.124:33211: connect: Connection refused (error 111)
I20260619 18:52:38.535120 26927 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.243.124" port: 33211 } attempt: 1
I20260619 18:52:38.536283 26930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:38.535780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.124:33211 (local address 127.0.0.1:53880)
0619 18:52:38.535998 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0619 18:52:38.536090 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.124:33211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260619 18:52:38.537240 26930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:38.536933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.124:33211 (local address 127.0.0.1:53896)
0619 18:52:38.537073 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 18:52:38.537139 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.124:33211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260619 18:52:38.537444 26927 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.243.124:33211: Network error: Client connection negotiation failed: client connection to 127.3.243.124:33211: connect: Connection refused (error 111)
W20260619 18:52:38.596570 27008 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 18:52:38.596909 27008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:52:38.597000 27008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:52:38.597114 27008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:52:38.604115 27008 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:52:38.604225 27008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:52:38.604316 27008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:52:38.604377 27008 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:52:38.604458 27008 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:52:38.605635 26927 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.243.124" port: 33211 } attempt: 2
I20260619 18:52:38.606673 26930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:38.606216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.124:33211 (local address 127.0.0.1:53906)
0619 18:52:38.606431 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0619 18:52:38.606519 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.124:33211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260619 18:52:38.607614 26930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:38.607300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.124:33211 (local address 127.0.0.1:53914)
0619 18:52:38.607437 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0619 18:52:38.607506 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.124:33211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260619 18:52:38.607797 26927 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.243.124:33211: Network error: Client connection negotiation failed: client connection to 127.3.243.124:33211: connect: Connection refused (error 111)
I20260619 18:52:38.614136 27008 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.84:44601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894810715282-4045-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894810715282-4045-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.243.126:38079,127.3.243.125:37293,127.3.243.124:33211
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894810715282-4045-0/krb5kdc/kudu_127.3.243.124.keytab
--principal=kudu/127.3.243.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.124:33211
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894810715282-4045-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.243.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894810715282-4045-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 17:49:28 UTC on ccycloud.quasar-jjuemm.root.comops.site
build id 4456547
ASAN enabled
I20260619 18:52:38.615361 27008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:52:38.616674 27008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:52:38.627874 27016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:52:38.627921 27014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:52:38.627949 27013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:52:38.628772 27008 server_base.cc:1034] running on GCE node
I20260619 18:52:38.629495 27008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:52:38.630452 27001 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.243.124" port: 33211 } attempt: 4
I20260619 18:52:38.630903 27008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:52:38.631577 27003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:38.631045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.124:33211 (local address 127.0.0.1:53922)
0619 18:52:38.631298 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0619 18:52:38.631397 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.124:33211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260619 18:52:38.632124 27008 hybrid_clock.cc:648] HybridClock initialized: now 1781895158632047 us; error 41 us; skew 500 ppm
I20260619 18:52:38.632525 27008 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 18:52:38.632591 27003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:38.632220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.124:33211 (local address 127.0.0.1:53928)
0619 18:52:38.632377 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 18:52:38.632464 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.124:33211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260619 18:52:38.632877 27001 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.243.124:33211: Network error: Client connection negotiation failed: client connection to 127.3.243.124:33211: connect: Connection refused (error 111)
Jun 19 18:52:38 dist-test-slave-m9wg krb5kdc[26847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895158, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:52:38.636574 27008 init.cc:377] Logged in from keytab as kudu/127.3.243.124@KRBTEST.COM (short username kudu)
I20260619 18:52:38.647212 27008 webserver.cc:460] Webserver started at http://127.3.243.124:42345/ using document root <none> and password file <none>
I20260619 18:52:38.647954 27008 fs_manager.cc:359] Metadata directory not provided
I20260619 18:52:38.648046 27008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:52:38.648386 27008 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:52:38.651360 27008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894810715282-4045-0/minicluster-data/master-2/data/instance:
uuid: "880045b44ae8492580bb907d77e97ea7"
format_stamp: "Formatted at 2026-06-19 18:52:38 on dist-test-slave-m9wg"
server_key: "d33c2c417084f3a2d95aa38a8b1ec5ca"
server_key_iv: "44f1316cac192d459973bc59f4c814ea"
server_key_version: "encryptionkey@0"
I20260619 18:52:38.652109 27008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894810715282-4045-0/minicluster-data/master-2/wal/instance:
uuid: "880045b44ae8492580bb907d77e97ea7"
format_stamp: "Formatted at 2026-06-19 18:52:38 on dist-test-slave-m9wg"
server_key: "d33c2c417084f3a2d95aa38a8b1ec5ca"
server_key_iv: "44f1316cac192d459973bc59f4c814ea"
server_key_version: "encryptionkey@0"
I20260619 18:52:38.657339 27008 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 18:52:38.660809 27023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:52:38.662096 27008 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 18:52:38.662250 27008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894810715282-4045-0/minicluster-data/master-2/data,/tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894810715282-4045-0/minicluster-data/master-2/wal
uuid: "880045b44ae8492580bb907d77e97ea7"
format_stamp: "Formatted at 2026-06-19 18:52:38 on dist-test-slave-m9wg"
server_key: "d33c2c417084f3a2d95aa38a8b1ec5ca"
server_key_iv: "44f1316cac192d459973bc59f4c814ea"
server_key_version: "encryptionkey@0"
I20260619 18:52:38.662436 27008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894810715282-4045-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894810715282-4045-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894810715282-4045-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:52:38.681782 27008 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:52:38.685519 27008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:52:38.685770 27008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:52:38.702696 27008 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.124:33211
I20260619 18:52:38.702715 27074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.124:33211 every 8 connection(s)
I20260619 18:52:38.704002 27008 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894810715282-4045-0/minicluster-data/master-2/data/info.pb
I20260619 18:52:38.707532 27075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:52:38.708110 4045 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu as pid 27008
I20260619 18:52:38.708308 4045 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRgbpDZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894810715282-4045-0/minicluster-data/master-2/wal/instance
I20260619 18:52:38.708719 4045 external_mini_cluster.cc:1428] Setting key f916066b5aaed988f37089a0a134efe0
I20260619 18:52:38.713985 26927 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.243.124" port: 33211 } attempt: 3
I20260619 18:52:38.716573 27075 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.243.126" port: 38079 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 18:52:38 dist-test-slave-m9wg krb5kdc[26847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895157, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.126@KRBTEST.COM for kudu/127.3.243.124@KRBTEST.COM
Jun 19 18:52:38 dist-test-slave-m9wg krb5kdc[26847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895157, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.124@KRBTEST.COM
Jun 19 18:52:38 dist-test-slave-m9wg krb5kdc[26847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895158, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.124@KRBTEST.COM for kudu/127.3.243.126@KRBTEST.COM
I20260619 18:52:38.732051 26930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:38.714563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.124:33211 (local address 127.0.0.1:53938)
0619 18:52:38.714873 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0619 18:52:38.714891 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 18:52:38.715006 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:52:38.716970 (+ 1964us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:52:38.716975 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:52:38.717438 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:52:38.717632 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:52:38.717644 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:52:38.719162 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:52:38.719167 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:52:38.720557 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:52:38.720573 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:52:38.721576 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:52:38.721581 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:52:38.721809 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:38.722326 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:52:38.722344 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:52:38.724649 (+ 2305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:52:38.729028 (+ 4379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:52:38.729034 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:52:38.729038 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:52:38.729515 (+ 477us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:52:38.729938 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:52:38.729961 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:52:38.729966 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:52:38.730139 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:52:38.730927 (+ 788us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:52:38.730933 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:52:38.731148 (+ 215us) client_negotiation.cc:769] Sending connection context
0619 18:52:38.731319 (+ 171us) client_negotiation.cc:240] Negotiation successful
0619 18:52:38.731634 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157}
I20260619 18:52:38.733310 27079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:38.714568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53938 (local address 127.3.243.124:33211)
0619 18:52:38.716625 (+ 2057us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:38.716630 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:38.716653 (+ 23us) server_negotiation.cc:407] Connection header received
0619 18:52:38.716696 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:38.716701 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:38.716783 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:38.716904 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:38.717839 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:38.719021 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:38.720682 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:38.721409 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:38.721518 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:38.724893 (+ 3375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:52:38.724904 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:52:38.724919 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:52:38.724964 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:52:38.728890 (+ 3926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:38.729655 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:38.729664 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:38.729671 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:38.729841 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:38.730247 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:38.730250 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:38.730252 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:38.730726 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:52:38.730923 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:38.731501 (+ 578us) server_negotiation.cc:299] Negotiation successful
0619 18:52:38.731953 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":1911,"thread_start_us":1743,"threads_started":1}
I20260619 18:52:38.734819 27078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:38.710389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53932 (local address 127.3.243.124:33211)
0619 18:52:38.711171 (+ 782us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:38.711185 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:38.711232 (+ 47us) server_negotiation.cc:407] Connection header received
0619 18:52:38.713547 (+ 2315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:38.713576 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:38.714103 (+ 527us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:38.714532 (+ 429us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:38.716674 (+ 2142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:38.718403 (+ 1729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:38.720090 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:38.721139 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:38.721313 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:38.725908 (+ 4595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:52:38.725915 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:52:38.725919 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:52:38.725957 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:52:38.729329 (+ 3372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:38.732012 (+ 2683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:38.732017 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:38.732021 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:38.732181 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:38.732606 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:38.732611 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:38.732614 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:38.732952 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:52:38.733185 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:38.734258 (+ 1073us) server_negotiation.cc:299] Negotiation successful
0619 18:52:38.734439 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":553,"thread_start_us":311,"threads_started":1}
I20260619 18:52:38.735198 26931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:38.719513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42530 (local address 127.3.243.126:38079)
0619 18:52:38.719684 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:38.719690 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:38.720326 (+ 636us) server_negotiation.cc:407] Connection header received
0619 18:52:38.720574 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:38.720579 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:38.720654 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:38.720756 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:38.722069 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:38.723109 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:38.724649 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:38.725159 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:38.725266 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:38.728756 (+ 3490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:52:38.728761 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:52:38.728764 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:52:38.728792 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:52:38.732381 (+ 3589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:38.733054 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:38.733060 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:38.733063 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:38.733183 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:38.733566 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:38.733570 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:38.733573 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:38.733875 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:52:38.733970 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:38.734541 (+ 571us) server_negotiation.cc:299] Negotiation successful
0619 18:52:38.734742 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 18:52:38.736168 27080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:38.719405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.126:38079 (local address 127.0.0.1:42530)
0619 18:52:38.720202 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0619 18:52:38.720244 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 18:52:38.720452 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:52:38.720822 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:52:38.720829 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:52:38.721669 (+ 840us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:52:38.721953 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:52:38.721966 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:52:38.723304 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:52:38.723310 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:52:38.724502 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:52:38.724517 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:52:38.725357 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:52:38.725364 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:52:38.725527 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:38.726084 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:52:38.726112 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:52:38.728580 (+ 2468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:52:38.732518 (+ 3938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:52:38.732529 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:52:38.732536 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:52:38.732937 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:52:38.733297 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:52:38.733301 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:52:38.733303 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:52:38.733441 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:52:38.734051 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:52:38.734058 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:52:38.734250 (+ 192us) client_negotiation.cc:769] Sending connection context
0619 18:52:38.734401 (+ 151us) client_negotiation.cc:240] Negotiation successful
0619 18:52:38.735828 (+ 1427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":432,"threads_started":1}
I20260619 18:52:38.737910 27075 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.243.125" port: 37293 } has no permanent_uuid. Determining permanent_uuid...
I20260619 18:52:38.740182 26927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 020a26e9047d461fb3be0ed825316f89: Bootstrap starting.
I20260619 18:52:38.743427 26927 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 020a26e9047d461fb3be0ed825316f89: Neither blocks nor log segments found. Creating new log.
Jun 19 18:52:38 dist-test-slave-m9wg krb5kdc[26847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895157, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.126@KRBTEST.COM
I20260619 18:52:38.744700 26927 log.cc:824] T 00000000000000000000000000000000 P 020a26e9047d461fb3be0ed825316f89: Log is configured to *not* fsync() on all Append() calls
I20260619 18:52:38.748225 26927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 020a26e9047d461fb3be0ed825316f89: No bootstrap required, opened a new log
Jun 19 18:52:38 dist-test-slave-m9wg krb5kdc[26847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895158, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.124@KRBTEST.COM for kudu/127.3.243.125@KRBTEST.COM
I20260619 18:52:38.752072 26931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:38.737066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42536 (local address 127.3.243.126:38079)
0619 18:52:38.737223 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:38.737228 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:38.737340 (+ 112us) server_negotiation.cc:407] Connection header received
0619 18:52:38.737511 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:38.737516 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:38.737586 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:38.737667 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:38.739073 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:38.739819 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:38.741076 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:38.741508 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:38.741620 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:38.745848 (+ 4228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:52:38.745852 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:52:38.745855 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:52:38.745883 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:52:38.749086 (+ 3203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:38.749731 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:38.749736 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:38.749740 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:38.749883 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:38.750247 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:38.750252 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:38.750255 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:38.750619 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:52:38.750902 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:38.751497 (+ 595us) server_negotiation.cc:299] Negotiation successful
0619 18:52:38.751705 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 18:52:38.756013 27080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:38.738653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.125:37293 (local address 127.0.0.1:48820)
0619 18:52:38.738889 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0619 18:52:38.738906 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 18:52:38.738993 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:52:38.742711 (+ 3718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:52:38.742716 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:52:38.743141 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:52:38.743310 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:52:38.743321 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:52:38.744662 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:52:38.744668 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:52:38.745930 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:52:38.745941 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:52:38.746877 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:52:38.746883 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:52:38.747054 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:38.747569 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:52:38.747591 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:52:38.750077 (+ 2486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:52:38.753187 (+ 3110us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:52:38.753192 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:52:38.753196 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:52:38.753665 (+ 469us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:52:38.754178 (+ 513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:52:38.754183 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:52:38.754187 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:52:38.754351 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:52:38.755060 (+ 709us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:52:38.755064 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:52:38.755237 (+ 173us) client_negotiation.cc:769] Sending connection context
0619 18:52:38.755381 (+ 144us) client_negotiation.cc:240] Negotiation successful
0619 18:52:38.755649 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133}
I20260619 18:52:38.756047 27005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:38.738639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48820 (local address 127.3.243.125:37293)
0619 18:52:38.742397 (+ 3758us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:38.742403 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:38.742423 (+ 20us) server_negotiation.cc:407] Connection header received
0619 18:52:38.742478 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:38.742483 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:38.742558 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:38.742669 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:38.743479 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:38.744493 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:38.746053 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:38.746660 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:38.746789 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:38.750285 (+ 3496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:52:38.750291 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:52:38.750296 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:52:38.750333 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:52:38.753064 (+ 2731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:38.753836 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:38.753842 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:38.753846 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:38.754029 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:38.754473 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:38.754478 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:38.754481 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:38.754821 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:52:38.754977 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:38.755481 (+ 504us) server_negotiation.cc:299] Negotiation successful
0619 18:52:38.755685 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3622}
I20260619 18:52:38.756978 27075 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.243.124" port: 33211 } has no permanent_uuid. Determining permanent_uuid...
I20260619 18:52:38.758085 26927 raft_consensus.cc:348] T 00000000000000000000000000000000 P 020a26e9047d461fb3be0ed825316f89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "020a26e9047d461fb3be0ed825316f89" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 38079 } } peers { permanent_uuid: "2b525a8d5f2247a689df2fbf1d4dcca7" member_type: VOTER last_known_addr { host: "127.3.243.125" port: 37293 } } peers { permanent_uuid: "880045b44ae8492580bb907d77e97ea7" member_type: VOTER last_known_addr { host: "127.3.243.124" port: 33211 } }
I20260619 18:52:38.758481 26927 raft_consensus.cc:729] T 00000000000000000000000000000000 P 020a26e9047d461fb3be0ed825316f89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 020a26e9047d461fb3be0ed825316f89, State: Initialized, Role: FOLLOWER
I20260619 18:52:38.758977 26927 consensus_queue.cc:260] T 00000000000000000000000000000000 P 020a26e9047d461fb3be0ed825316f89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "020a26e9047d461fb3be0ed825316f89" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 38079 } } peers { permanent_uuid: "2b525a8d5f2247a689df2fbf1d4dcca7" member_type: VOTER last_known_addr { host: "127.3.243.125" port: 37293 } } peers { permanent_uuid: "880045b44ae8492580bb907d77e97ea7" member_type: VOTER last_known_addr { host: "127.3.243.124" port: 33211 } }
I20260619 18:52:38.759920 27083 sys_catalog.cc:455] T 00000000000000000000000000000000 P 020a26e9047d461fb3be0ed825316f89 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "020a26e9047d461fb3be0ed825316f89" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 38079 } } peers { permanent_uuid: "2b525a8d5f2247a689df2fbf1d4dcca7" member_type: VOTER last_known_addr { host: "127.3.243.125" port: 37293 } } peers { permanent_uuid: "880045b44ae8492580bb907d77e97ea7" member_type: VOTER last_known_addr { host: "127.3.243.124" port: 33211 } } }
I20260619 18:52:38.760428 27083 sys_catalog.cc:458] T 00000000000000000000000000000000 P 020a26e9047d461fb3be0ed825316f89 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:52:38.761233 26927 sys_catalog.cc:564] T 00000000000000000000000000000000 P 020a26e9047d461fb3be0ed825316f89 [sys.catalog]: configured and running, proceeding with master startup.
Jun 19 18:52:38 dist-test-slave-m9wg krb5kdc[26847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895158, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.124@KRBTEST.COM for kudu/127.3.243.124@KRBTEST.COM
Jun 19 18:52:38 dist-test-slave-m9wg krb5kdc[26847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895157, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.125@KRBTEST.COM
I20260619 18:52:38.771627 27080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:38.757743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.124:33211 (local address 127.0.0.1:53952)
0619 18:52:38.757959 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0619 18:52:38.757973 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 18:52:38.758065 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:52:38.758304 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:52:38.758307 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:52:38.758722 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:52:38.758869 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:52:38.758877 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:52:38.760053 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:52:38.760059 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:52:38.761101 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:52:38.761112 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:52:38.761853 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:52:38.761859 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:52:38.762019 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:38.762539 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:52:38.762561 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:52:38.765171 (+ 2610us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:52:38.768930 (+ 3759us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:52:38.768936 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:52:38.768940 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:52:38.769424 (+ 484us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:52:38.769836 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:52:38.769840 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:52:38.769844 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:52:38.769978 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:52:38.770636 (+ 658us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:52:38.770641 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:52:38.770794 (+ 153us) client_negotiation.cc:769] Sending connection context
0619 18:52:38.770934 (+ 140us) client_negotiation.cc:240] Negotiation successful
0619 18:52:38.771243 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137}
I20260619 18:52:38.772779 27078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:38.757733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53952 (local address 127.3.243.124:33211)
0619 18:52:38.757889 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:38.757895 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:38.758031 (+ 136us) server_negotiation.cc:407] Connection header received
0619 18:52:38.758146 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:38.758151 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:38.758206 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:38.758284 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:38.759006 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:38.759904 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:38.761246 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:38.761705 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:38.761812 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:38.765366 (+ 3554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:52:38.765372 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:52:38.765378 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:52:38.765413 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:52:38.768791 (+ 3378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:38.769561 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:38.769566 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:38.769570 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:38.769734 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:38.770095 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:38.770099 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:38.770103 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:38.770518 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:52:38.772037 (+ 1519us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:38.772197 (+ 160us) server_negotiation.cc:299] Negotiation successful
0619 18:52:38.772435 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 18:52:38.774187 27082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:38.754979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48822 (local address 127.3.243.125:37293)
0619 18:52:38.755531 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:38.755537 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:38.755559 (+ 22us) server_negotiation.cc:407] Connection header received
0619 18:52:38.755605 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:38.755611 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:38.755676 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:38.757809 (+ 2133us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:38.758063 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:38.759720 (+ 1657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:38.761411 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:38.762016 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:38.762137 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:38.767286 (+ 5149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:52:38.767293 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:52:38.767297 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:52:38.767333 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:52:38.770623 (+ 3290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:38.771383 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:38.771388 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:38.771392 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:38.771520 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:38.771951 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:38.771956 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:38.771960 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:38.772296 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:52:38.773453 (+ 1157us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:38.773571 (+ 118us) server_negotiation.cc:299] Negotiation successful
0619 18:52:38.773801 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":230,"threads_started":1}
I20260619 18:52:38.778892 4045 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:52:38.778977 27075 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 880045b44ae8492580bb907d77e97ea7: Bootstrap starting.
I20260619 18:52:38.781227 26902 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "020a26e9047d461fb3be0ed825316f89"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42536
I20260619 18:52:38.781571 26902 raft_consensus.cc:482] T 00000000000000000000000000000000 P 020a26e9047d461fb3be0ed825316f89 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 18:52:38.781747 26902 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 020a26e9047d461fb3be0ed825316f89 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:52:38.783238 27075 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 880045b44ae8492580bb907d77e97ea7: Neither blocks nor log segments found. Creating new log.
I20260619 18:52:38.784107 26902 raft_consensus.cc:504] T 00000000000000000000000000000000 P 020a26e9047d461fb3be0ed825316f89 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "020a26e9047d461fb3be0ed825316f89" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 38079 } } peers { permanent_uuid: "2b525a8d5f2247a689df2fbf1d4dcca7" member_type: VOTER last_known_addr { host: "127.3.243.125" port: 37293 } } peers { permanent_uuid: "880045b44ae8492580bb907d77e97ea7" member_type: VOTER last_known_addr { host: "127.3.243.124" port: 33211 } }
I20260619 18:52:38.784816 27075 log.cc:824] T 00000000000000000000000000000000 P 880045b44ae8492580bb907d77e97ea7: Log is configured to *not* fsync() on all Append() calls
I20260619 18:52:38.786391 26902 leader_election.cc:288] T 00000000000000000000000000000000 P 020a26e9047d461fb3be0ed825316f89 [CANDIDATE]: Term 1 election: Requested vote from peers 2b525a8d5f2247a689df2fbf1d4dcca7 (127.3.243.125:37293), 880045b44ae8492580bb907d77e97ea7 (127.3.243.124:33211)
I20260619 18:52:38.788625 27075 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 880045b44ae8492580bb907d77e97ea7: No bootstrap required, opened a new log
I20260619 18:52:38.800072 27075 raft_consensus.cc:348] T 00000000000000000000000000000000 P 880045b44ae8492580bb907d77e97ea7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "020a26e9047d461fb3be0ed825316f89" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 38079 } } peers { permanent_uuid: "2b525a8d5f2247a689df2fbf1d4dcca7" member_type: VOTER last_known_addr { host: "127.3.243.125" port: 37293 } } peers { permanent_uuid: "880045b44ae8492580bb907d77e97ea7" member_type: VOTER last_known_addr { host: "127.3.243.124" port: 33211 } }
I20260619 18:52:38.800580 27075 raft_consensus.cc:729] T 00000000000000000000000000000000 P 880045b44ae8492580bb907d77e97ea7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 880045b44ae8492580bb907d77e97ea7, State: Initialized, Role: FOLLOWER
I20260619 18:52:38.801141 27075 consensus_queue.cc:260] T 00000000000000000000000000000000 P 880045b44ae8492580bb907d77e97ea7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "020a26e9047d461fb3be0ed825316f89" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 38079 } } peers { permanent_uuid: "2b525a8d5f2247a689df2fbf1d4dcca7" member_type: VOTER last_known_addr { host: "127.3.243.125" port: 37293 } } peers { permanent_uuid: "880045b44ae8492580bb907d77e97ea7" member_type: VOTER last_known_addr { host: "127.3.243.124" port: 33211 } }
I20260619 18:52:38.802563 27098 sys_catalog.cc:455] T 00000000000000000000000000000000 P 880045b44ae8492580bb907d77e97ea7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "020a26e9047d461fb3be0ed825316f89" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 38079 } } peers { permanent_uuid: "2b525a8d5f2247a689df2fbf1d4dcca7" member_type: VOTER last_known_addr { host: "127.3.243.125" port: 37293 } } peers { permanent_uuid: "880045b44ae8492580bb907d77e97ea7" member_type: VOTER last_known_addr { host: "127.3.243.124" port: 33211 } } }
I20260619 18:52:38.803613 27075 sys_catalog.cc:564] T 00000000000000000000000000000000 P 880045b44ae8492580bb907d77e97ea7 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:52:38.803634 27050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "020a26e9047d461fb3be0ed825316f89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "880045b44ae8492580bb907d77e97ea7"
I20260619 18:52:38.804294 27050 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 880045b44ae8492580bb907d77e97ea7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:52:38.807950 27050 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 880045b44ae8492580bb907d77e97ea7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 020a26e9047d461fb3be0ed825316f89 in term 1.
I20260619 18:52:38.808924 26877 leader_election.cc:302] T 00000000000000000000000000000000 P 020a26e9047d461fb3be0ed825316f89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 020a26e9047d461fb3be0ed825316f89, 880045b44ae8492580bb907d77e97ea7; no voters:
I20260619 18:52:38.809466 27083 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 020a26e9047d461fb3be0ed825316f89 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:52:38.803447 27098 sys_catalog.cc:458] T 00000000000000000000000000000000 P 880045b44ae8492580bb907d77e97ea7 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:52:38.837915 27083 raft_consensus.cc:686] T 00000000000000000000000000000000 P 020a26e9047d461fb3be0ed825316f89 [term 1 LEADER]: Becoming Leader. State: Replica: 020a26e9047d461fb3be0ed825316f89, State: Running, Role: LEADER
I20260619 18:52:38.841176 27083 consensus_queue.cc:237] T 00000000000000000000000000000000 P 020a26e9047d461fb3be0ed825316f89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "020a26e9047d461fb3be0ed825316f89" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 38079 } } peers { permanent_uuid: "2b525a8d5f2247a689df2fbf1d4dcca7" member_type: VOTER last_known_addr { host: "127.3.243.125" port: 37293 } } peers { permanent_uuid: "880045b44ae8492580bb907d77e97ea7" member_type: VOTER last_known_addr { host: "127.3.243.124" port: 33211 } }
I20260619 18:52:38.852815 27102 sys_catalog.cc:455] T 00000000000000000000000000000000 P 020a26e9047d461fb3be0ed825316f89 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 020a26e9047d461fb3be0ed825316f89. Latest consensus state: current_term: 1 leader_uuid: "020a26e9047d461fb3be0ed825316f89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "020a26e9047d461fb3be0ed825316f89" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 38079 } } peers { permanent_uuid: "2b525a8d5f2247a689df2fbf1d4dcca7" member_type: VOTER last_known_addr { host: "127.3.243.125" port: 37293 } } peers { permanent_uuid: "880045b44ae8492580bb907d77e97ea7" member_type: VOTER last_known_addr { host: "127.3.243.124" port: 33211 } } }
I20260619 18:52:38.854094 27102 sys_catalog.cc:458] T 00000000000000000000000000000000 P 020a26e9047d461fb3be0ed825316f89 [sys.catalog]: This master's current role is: LEADER
I20260619 18:52:38.862844 27078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:38.811188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53966 (local address 127.3.243.124:33211)
0619 18:52:38.811386 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:38.811394 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:38.842340 (+ 30946us) server_negotiation.cc:407] Connection header received
0619 18:52:38.842734 (+ 394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:38.842741 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:38.842831 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:38.842904 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:38.846591 (+ 3687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:38.847760 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:38.850030 (+ 2270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:38.850689 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:38.850807 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:38.854450 (+ 3643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:52:38.854457 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:52:38.854462 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:52:38.854503 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:52:38.858365 (+ 3862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:38.859603 (+ 1238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:38.859609 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:38.859613 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:38.859789 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:38.860303 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:38.860309 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:38.860312 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:38.860962 (+ 650us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:52:38.861232 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:38.862213 (+ 981us) server_negotiation.cc:299] Negotiation successful
0619 18:52:38.862464 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 18:52:38.866228 27082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:38.804482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48832 (local address 127.3.243.125:37293)
0619 18:52:38.804695 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:38.804701 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:38.839845 (+ 35144us) server_negotiation.cc:407] Connection header received
0619 18:52:38.841677 (+ 1832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:38.841684 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:38.841797 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:38.841882 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:38.846791 (+ 4909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:38.847832 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:38.852849 (+ 5017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:38.853546 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:38.853644 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:38.858533 (+ 4889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:52:38.858541 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:52:38.858547 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:52:38.858592 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:52:38.862078 (+ 3486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:38.863391 (+ 1313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:38.863398 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:38.863402 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:38.863560 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:38.864193 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:38.864198 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:38.864202 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:38.864569 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:52:38.864803 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:38.865632 (+ 829us) server_negotiation.cc:299] Negotiation successful
0619 18:52:38.865878 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260619 18:52:38.880795 27110 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 020a26e9047d461fb3be0ed825316f89: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 18:52:38.881094 27110 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 18:52:38.881328 27111 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:52:38.883410 27111 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:52:38.873672 26860 master_runner.cc:186] Error getting master registration for 127.3.243.125:37293: OK, instance_id { permanent_uuid: "2b525a8d5f2247a689df2fbf1d4dcca7" instance_seqno: 1781895158213244 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 18:52:38.887889 26931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:38.812867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42552 (local address 127.3.243.126:38079)
0619 18:52:38.813032 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:38.813038 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:38.841792 (+ 28754us) server_negotiation.cc:407] Connection header received
0619 18:52:38.841972 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:38.841977 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:38.842058 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:38.842147 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:38.844238 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:38.845285 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:38.847067 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:38.847728 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:38.847825 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:38.867637 (+ 19812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:52:38.867645 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:52:38.867651 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:52:38.867699 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:52:38.871331 (+ 3632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:38.872603 (+ 1272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:38.872609 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:38.872612 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:38.872816 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:38.880600 (+ 7784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:38.880606 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:38.880614 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:38.881074 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:52:38.881372 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:38.887169 (+ 5797us) server_negotiation.cc:299] Negotiation successful
0619 18:52:38.887502 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 18:52:38.888731 26860 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 18:52:38.895788 27050 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 880045b44ae8492580bb907d77e97ea7 [term 1 FOLLOWER]: Refusing update from remote peer 020a26e9047d461fb3be0ed825316f89: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:52:38.896776 27102 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 020a26e9047d461fb3be0ed825316f89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "880045b44ae8492580bb907d77e97ea7" member_type: VOTER last_known_addr { host: "127.3.243.124" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:52:38.907037 27098 sys_catalog.cc:455] T 00000000000000000000000000000000 P 880045b44ae8492580bb907d77e97ea7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 020a26e9047d461fb3be0ed825316f89. Latest consensus state: current_term: 1 leader_uuid: "020a26e9047d461fb3be0ed825316f89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "020a26e9047d461fb3be0ed825316f89" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 38079 } } peers { permanent_uuid: "2b525a8d5f2247a689df2fbf1d4dcca7" member_type: VOTER last_known_addr { host: "127.3.243.125" port: 37293 } } peers { permanent_uuid: "880045b44ae8492580bb907d77e97ea7" member_type: VOTER last_known_addr { host: "127.3.243.124" port: 33211 } } }
I20260619 18:52:38.907331 27098 sys_catalog.cc:458] T 00000000000000000000000000000000 P 880045b44ae8492580bb907d77e97ea7 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:52:38.910315 27116 mvcc.cc:205] Tried to move back new op lower bound from 7298642570827259904 to 7298642570631110656. Current Snapshot: MvccSnapshot[applied={T|T < 7298642570827259904}]
I20260619 18:52:38.916281 27120 mvcc.cc:205] Tried to move back new op lower bound from 7298642570827259904 to 7298642570631110656. Current Snapshot: MvccSnapshot[applied={T|T < 7298642570827259904}]
I20260619 18:52:38.908034 27008 master_runner.cc:186] Error getting master registration for 127.3.243.125:37293: OK, instance_id { permanent_uuid: "2b525a8d5f2247a689df2fbf1d4dcca7" instance_seqno: 1781895158213244 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 18:52:38.918926 27008 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 18:52:38.919624 27102 sys_catalog.cc:455] T 00000000000000000000000000000000 P 020a26e9047d461fb3be0ed825316f89 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "020a26e9047d461fb3be0ed825316f89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "020a26e9047d461fb3be0ed825316f89" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 38079 } } peers { permanent_uuid: "2b525a8d5f2247a689df2fbf1d4dcca7" member_type: VOTER last_known_addr { host: "127.3.243.125" port: 37293 } } peers { permanent_uuid: "880045b44ae8492580bb907d77e97ea7" member_type: VOTER last_known_addr { host: "127.3.243.124" port: 33211 } } }
I20260619 18:52:38.920221 27102 sys_catalog.cc:458] T 00000000000000000000000000000000 P 020a26e9047d461fb3be0ed825316f89 [sys.catalog]: This master's current role is: LEADER
I20260619 18:52:38.920692 27098 sys_catalog.cc:455] T 00000000000000000000000000000000 P 880045b44ae8492580bb907d77e97ea7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "020a26e9047d461fb3be0ed825316f89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "020a26e9047d461fb3be0ed825316f89" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 38079 } } peers { permanent_uuid: "2b525a8d5f2247a689df2fbf1d4dcca7" member_type: VOTER last_known_addr { host: "127.3.243.125" port: 37293 } } peers { permanent_uuid: "880045b44ae8492580bb907d77e97ea7" member_type: VOTER last_known_addr { host: "127.3.243.124" port: 33211 } } }
I20260619 18:52:38.920923 27098 sys_catalog.cc:458] T 00000000000000000000000000000000 P 880045b44ae8492580bb907d77e97ea7 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:52:38.923552 27111 catalog_manager.cc:1347] Generated new cluster ID: e492603c31ab48fbb39497e3f6e66023
I20260619 18:52:38.923653 27111 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
W20260619 18:52:38.926311 27119 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 880045b44ae8492580bb907d77e97ea7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 18:52:38.926442 27119 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 18:52:38.932029 27001 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.243.124" port: 33211 } attempt: 5
Jun 19 18:52:38 dist-test-slave-m9wg krb5kdc[26847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781895158, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.125@KRBTEST.COM for kudu/127.3.243.124@KRBTEST.COM
I20260619 18:52:38.950553 27003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:38.932587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.124:33211 (local address 127.0.0.1:53982)
0619 18:52:38.932933 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0619 18:52:38.932954 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 18:52:38.933064 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:52:38.933895 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:52:38.933901 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:52:38.934439 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:52:38.934643 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:52:38.934656 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:52:38.936125 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:52:38.936132 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:52:38.937638 (+ 1506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:52:38.937654 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:52:38.939942 (+ 2288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:52:38.939949 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:52:38.940195 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:38.940837 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:52:38.940862 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:52:38.943637 (+ 2775us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:52:38.947654 (+ 4017us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:52:38.947661 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:52:38.947665 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:52:38.948196 (+ 531us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:52:38.948647 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:52:38.948652 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:52:38.948656 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:52:38.948819 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:52:38.949536 (+ 717us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:52:38.949542 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:52:38.949713 (+ 171us) client_negotiation.cc:769] Sending connection context
0619 18:52:38.949860 (+ 147us) client_negotiation.cc:240] Negotiation successful
0619 18:52:38.950156 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221}
I20260619 18:52:38.951524 27078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:38.933257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53982 (local address 127.3.243.124:33211)
0619 18:52:38.933421 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:38.933427 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:38.933446 (+ 19us) server_negotiation.cc:407] Connection header received
0619 18:52:38.933497 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:38.933502 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:38.933596 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:38.933693 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:38.934814 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:38.935965 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:38.938929 (+ 2964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:38.939608 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:38.939732 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:38.943858 (+ 4126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:52:38.943865 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:52:38.943870 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:52:38.943910 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:52:38.947492 (+ 3582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:38.948342 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:38.948347 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:38.948351 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:38.948513 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:52:38.948949 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:52:38.948954 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:52:38.948958 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:52:38.949320 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:52:38.949472 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:38.950912 (+ 1440us) server_negotiation.cc:299] Negotiation successful
0619 18:52:38.951142 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 18:52:38.957285 27001 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b525a8d5f2247a689df2fbf1d4dcca7: Bootstrap starting.
I20260619 18:52:38.962466 27111 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:52:38.964401 27111 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:52:38.968945 27001 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b525a8d5f2247a689df2fbf1d4dcca7: Neither blocks nor log segments found. Creating new log.
I20260619 18:52:38.971474 27001 log.cc:824] T 00000000000000000000000000000000 P 2b525a8d5f2247a689df2fbf1d4dcca7: Log is configured to *not* fsync() on all Append() calls
I20260619 18:52:38.980543 27111 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 020a26e9047d461fb3be0ed825316f89: Generated new TSK 0
I20260619 18:52:38.983112 27111 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:52:38.989560 27001 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b525a8d5f2247a689df2fbf1d4dcca7: No bootstrap required, opened a new log
I20260619 18:52:39.007850 27001 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b525a8d5f2247a689df2fbf1d4dcca7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "020a26e9047d461fb3be0ed825316f89" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 38079 } } peers { permanent_uuid: "2b525a8d5f2247a689df2fbf1d4dcca7" member_type: VOTER last_known_addr { host: "127.3.243.125" port: 37293 } } peers { permanent_uuid: "880045b44ae8492580bb907d77e97ea7" member_type: VOTER last_known_addr { host: "127.3.243.124" port: 33211 } }
I20260619 18:52:39.008759 27001 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b525a8d5f2247a689df2fbf1d4dcca7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b525a8d5f2247a689df2fbf1d4dcca7, State: Initialized, Role: FOLLOWER
I20260619 18:52:39.009299 27001 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b525a8d5f2247a689df2fbf1d4dcca7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "020a26e9047d461fb3be0ed825316f89" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 38079 } } peers { permanent_uuid: "2b525a8d5f2247a689df2fbf1d4dcca7" member_type: VOTER last_known_addr { host: "127.3.243.125" port: 37293 } } peers { permanent_uuid: "880045b44ae8492580bb907d77e97ea7" member_type: VOTER last_known_addr { host: "127.3.243.124" port: 33211 } }
I20260619 18:52:39.013026 27126 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b525a8d5f2247a689df2fbf1d4dcca7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "020a26e9047d461fb3be0ed825316f89" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 38079 } } peers { permanent_uuid: "2b525a8d5f2247a689df2fbf1d4dcca7" member_type: VOTER last_known_addr { host: "127.3.243.125" port: 37293 } } peers { permanent_uuid: "880045b44ae8492580bb907d77e97ea7" member_type: VOTER last_known_addr { host: "127.3.243.124" port: 33211 } } }
I20260619 18:52:39.013818 27126 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b525a8d5f2247a689df2fbf1d4dcca7 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:52:39.014354 27001 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b525a8d5f2247a689df2fbf1d4dcca7 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:52:39.014784 26975 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b525a8d5f2247a689df2fbf1d4dcca7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:52:39.018597 26975 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2b525a8d5f2247a689df2fbf1d4dcca7 [term 1 FOLLOWER]: Refusing update from remote peer 020a26e9047d461fb3be0ed825316f89: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:52:39.019757 27083 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 020a26e9047d461fb3be0ed825316f89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b525a8d5f2247a689df2fbf1d4dcca7" member_type: VOTER last_known_addr { host: "127.3.243.125" port: 37293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:52:39.020278 26976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "020a26e9047d461fb3be0ed825316f89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "2b525a8d5f2247a689df2fbf1d4dcca7"
I20260619 18:52:39.023665 26976 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2b525a8d5f2247a689df2fbf1d4dcca7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 020a26e9047d461fb3be0ed825316f89 in term 1.
I20260619 18:52:39.065817 27126 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b525a8d5f2247a689df2fbf1d4dcca7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 020a26e9047d461fb3be0ed825316f89. Latest consensus state: current_term: 1 leader_uuid: "020a26e9047d461fb3be0ed825316f89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "020a26e9047d461fb3be0ed825316f89" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 38079 } } peers { permanent_uuid: "2b525a8d5f2247a689df2fbf1d4dcca7" member_type: VOTER last_known_addr { host: "127.3.243.125" port: 37293 } } peers { permanent_uuid: "880045b44ae8492580bb907d77e97ea7" member_type: VOTER last_known_addr { host: "127.3.243.124" port: 33211 } } }
I20260619 18:52:39.066226 27126 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b525a8d5f2247a689df2fbf1d4dcca7 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:52:39.065879 27134 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b525a8d5f2247a689df2fbf1d4dcca7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "020a26e9047d461fb3be0ed825316f89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "020a26e9047d461fb3be0ed825316f89" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 38079 } } peers { permanent_uuid: "2b525a8d5f2247a689df2fbf1d4dcca7" member_type: VOTER last_known_addr { host: "127.3.243.125" port: 37293 } } peers { permanent_uuid: "880045b44ae8492580bb907d77e97ea7" member_type: VOTER last_known_addr { host: "127.3.243.124" port: 33211 } } }
I20260619 18:52:39.066349 27134 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b525a8d5f2247a689df2fbf1d4dcca7 [sys.catalog]: This master's current role is: FOLLOWER
W20260619 18:52:39.119288 27082 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:52:39.109230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48834 (local address 127.3.243.125:37293)
0619 18:52:39.109411 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:39.109417 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:39.116536 (+ 7119us) server_negotiation.cc:407] Connection header received
0619 18:52:39.117504 (+ 968us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:39.117512 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:39.117636 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:39.117715 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:39.118989 (+ 1274us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48834: BlockingRecv error: recv got EOF from 127.0.0.1:48834 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260619 18:52:39.121963 26931 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:52:39.114073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42564 (local address 127.3.243.126:38079)
0619 18:52:39.114346 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:39.114352 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:39.117373 (+ 3021us) server_negotiation.cc:407] Connection header received
0619 18:52:39.120303 (+ 2930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:39.120311 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:39.120400 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:39.120479 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:39.121686 (+ 1207us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42564: BlockingRecv error: recv got EOF from 127.0.0.1:42564 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
W20260619 18:52:39.122145 27078 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:52:39.108141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53996 (local address 127.3.243.124:33211)
0619 18:52:39.108371 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:39.108378 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:39.112730 (+ 4352us) server_negotiation.cc:407] Connection header received
0619 18:52:39.120457 (+ 7727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:39.120464 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:39.120548 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:39.120617 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:39.121868 (+ 1251us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53996: BlockingRecv error: recv got EOF from 127.0.0.1:53996 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260619 18:52:39.144464 27082 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:52:39.141130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48840 (local address 127.3.243.125:37293)
0619 18:52:39.141322 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:39.141328 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:39.142623 (+ 1295us) server_negotiation.cc:407] Connection header received
0619 18:52:39.142952 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:39.142957 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:39.143106 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:39.143175 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:39.144230 (+ 1055us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48840: BlockingRecv error: recv got EOF from 127.0.0.1:48840 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 18:52:39.163985 27154 catalog_manager.cc:1259] Loaded cluster ID: e492603c31ab48fbb39497e3f6e66023
I20260619 18:52:39.164109 27154 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2b525a8d5f2247a689df2fbf1d4dcca7: loading cluster ID for follower catalog manager: success
I20260619 18:52:39.168128 27154 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2b525a8d5f2247a689df2fbf1d4dcca7: acquiring CA information for follower catalog manager: success
I20260619 18:52:39.170655 27154 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2b525a8d5f2247a689df2fbf1d4dcca7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 18:52:39.205803 27082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:39.186082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48848 (local address 127.3.243.125:37293)
0619 18:52:39.186328 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:39.186334 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:39.198075 (+ 11741us) server_negotiation.cc:407] Connection header received
0619 18:52:39.198325 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:39.198332 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:39.198528 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:39.198663 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:52:39.199156 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:39.200350 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:39.202327 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:39.203039 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:39.203167 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:39.203732 (+ 565us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:52:39.204446 (+ 714us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:52:39.204585 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:39.205349 (+ 764us) server_negotiation.cc:299] Negotiation successful
0619 18:52:39.205439 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260619 18:52:39.256690 27078 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:52:39.251131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54012 (local address 127.3.243.124:33211)
0619 18:52:39.251353 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:39.251359 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:39.254842 (+ 3483us) server_negotiation.cc:407] Connection header received
0619 18:52:39.255116 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:39.255123 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:39.255250 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:39.255352 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:39.256450 (+ 1098us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54012: BlockingRecv error: recv got EOF from 127.0.0.1:54012 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 18:52:39.287225 27082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:39.263379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48862 (local address 127.3.243.125:37293)
0619 18:52:39.263553 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:39.263559 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:39.273364 (+ 9805us) server_negotiation.cc:407] Connection header received
0619 18:52:39.280403 (+ 7039us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:39.280410 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:39.280529 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:39.280601 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:52:39.281336 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:39.282493 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:39.284573 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:39.285251 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:39.285372 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:39.286051 (+ 679us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:52:39.286198 (+ 147us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:52:39.286255 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:39.286819 (+ 564us) server_negotiation.cc:299] Negotiation successful
0619 18:52:39.286907 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260619 18:52:39.335590 27078 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:52:39.321679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54016 (local address 127.3.243.124:33211)
0619 18:52:39.321900 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:39.321906 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:39.333487 (+ 11581us) server_negotiation.cc:407] Connection header received
0619 18:52:39.334079 (+ 592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:39.334086 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:39.334204 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:39.334300 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:39.335345 (+ 1045us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54016: BlockingRecv error: recv got EOF from 127.0.0.1:54016 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
I20260619 18:52:39.387405 27082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:39.372786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48868 (local address 127.3.243.125:37293)
0619 18:52:39.373031 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:39.373037 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:39.380166 (+ 7129us) server_negotiation.cc:407] Connection header received
0619 18:52:39.380731 (+ 565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:39.380738 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:39.380845 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:39.380921 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:52:39.381646 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:39.382845 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:39.384873 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:39.385545 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:39.385649 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:39.386313 (+ 664us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:52:39.386460 (+ 147us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:52:39.386525 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:39.386900 (+ 375us) server_negotiation.cc:299] Negotiation successful
0619 18:52:39.387011 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
W20260619 18:52:39.435899 27078 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:52:39.421555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54026 (local address 127.3.243.124:33211)
0619 18:52:39.421747 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:39.421754 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:39.433869 (+ 12115us) server_negotiation.cc:407] Connection header received
0619 18:52:39.434404 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:39.434411 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:39.434506 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:39.434580 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:39.435642 (+ 1062us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54026: BlockingRecv error: recv got EOF from 127.0.0.1:54026 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 18:52:39.483896 27082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:39.477901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48878 (local address 127.3.243.125:37293)
0619 18:52:39.478121 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:39.478126 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:39.478442 (+ 316us) server_negotiation.cc:407] Connection header received
0619 18:52:39.478675 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:39.478679 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:39.478797 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:39.478883 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:52:39.479390 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:39.480336 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:39.482225 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:39.482689 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:39.482792 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:39.483259 (+ 467us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:52:39.483377 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:52:39.483443 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:39.483605 (+ 162us) server_negotiation.cc:299] Negotiation successful
0619 18:52:39.483671 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
W20260619 18:52:39.491806 27078 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:52:39.489983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54038 (local address 127.3.243.124:33211)
0619 18:52:39.490190 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:39.490194 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:39.490453 (+ 259us) server_negotiation.cc:407] Connection header received
0619 18:52:39.490620 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:39.490624 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:39.490691 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:39.490787 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:39.491548 (+ 761us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54038: BlockingRecv error: recv got EOF from 127.0.0.1:54038 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
I20260619 18:52:39.517653 27082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:39.511708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48888 (local address 127.3.243.125:37293)
0619 18:52:39.511861 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:39.511865 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:39.512288 (+ 423us) server_negotiation.cc:407] Connection header received
0619 18:52:39.512502 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:39.512506 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:39.512575 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:39.512659 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:52:39.513137 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:39.513987 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:39.515920 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:39.516423 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:39.516516 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:39.517004 (+ 488us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:52:39.517113 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:52:39.517180 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:39.517335 (+ 155us) server_negotiation.cc:299] Negotiation successful
0619 18:52:39.517401 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
W20260619 18:52:39.525148 27078 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:52:39.523335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54042 (local address 127.3.243.124:33211)
0619 18:52:39.523503 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:39.523509 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:39.523849 (+ 340us) server_negotiation.cc:407] Connection header received
0619 18:52:39.524019 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:39.524023 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:39.524096 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:39.524198 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:39.524934 (+ 736us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54042: BlockingRecv error: recv got EOF from 127.0.0.1:54042 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 18:52:39.566439 27082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:39.560281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48890 (local address 127.3.243.125:37293)
0619 18:52:39.560514 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:39.560521 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:39.560942 (+ 421us) server_negotiation.cc:407] Connection header received
0619 18:52:39.561130 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:39.561134 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:39.561222 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:39.561318 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:52:39.561836 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:39.562600 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:39.564604 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:39.565018 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:39.565107 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:39.565659 (+ 552us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:52:39.565820 (+ 161us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:52:39.565891 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:39.566055 (+ 164us) server_negotiation.cc:299] Negotiation successful
0619 18:52:39.566132 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260619 18:52:39.574012 27078 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:52:39.572200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54054 (local address 127.3.243.124:33211)
0619 18:52:39.572411 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:39.572417 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:39.572654 (+ 237us) server_negotiation.cc:407] Connection header received
0619 18:52:39.572779 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:39.572785 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:39.572884 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:39.573000 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:39.573768 (+ 768us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54054: BlockingRecv error: recv got EOF from 127.0.0.1:54054 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 18:52:39.610886 27083 sys_catalog.cc:455] T 00000000000000000000000000000000 P 020a26e9047d461fb3be0ed825316f89 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "020a26e9047d461fb3be0ed825316f89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "020a26e9047d461fb3be0ed825316f89" member_type: VOTER last_known_addr { host: "127.3.243.126" port: 38079 } } peers { permanent_uuid: "2b525a8d5f2247a689df2fbf1d4dcca7" member_type: VOTER last_known_addr { host: "127.3.243.125" port: 37293 } } peers { permanent_uuid: "880045b44ae8492580bb907d77e97ea7" member_type: VOTER last_known_addr { host: "127.3.243.124" port: 33211 } } }
I20260619 18:52:39.611219 27083 sys_catalog.cc:458] T 00000000000000000000000000000000 P 020a26e9047d461fb3be0ed825316f89 [sys.catalog]: This master's current role is: LEADER
I20260619 18:52:39.647712 27082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:39.641538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48896 (local address 127.3.243.125:37293)
0619 18:52:39.641734 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:39.641741 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:39.641840 (+ 99us) server_negotiation.cc:407] Connection header received
0619 18:52:39.642089 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:39.642094 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:39.642324 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:39.642421 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:52:39.642865 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:39.644117 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:39.645724 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:39.646284 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:39.646537 (+ 253us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:39.647010 (+ 473us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:52:39.647144 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:52:39.647226 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:39.647365 (+ 139us) server_negotiation.cc:299] Negotiation successful
0619 18:52:39.647445 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260619 18:52:39.655191 27078 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:52:39.653282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54066 (local address 127.3.243.124:33211)
0619 18:52:39.653469 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:39.653475 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:39.653750 (+ 275us) server_negotiation.cc:407] Connection header received
0619 18:52:39.653914 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:39.653920 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:39.654026 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:39.654132 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:39.654962 (+ 830us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54066: BlockingRecv error: recv got EOF from 127.0.0.1:54066 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 18:52:39.791494 27082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:39.786338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47636 (local address 127.3.243.125:37293)
0619 18:52:39.786593 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:39.786599 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:39.786877 (+ 278us) server_negotiation.cc:407] Connection header received
0619 18:52:39.787093 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:39.787098 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:39.787168 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:39.787244 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:52:39.787773 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:39.788541 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:39.789926 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:39.790355 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:39.790461 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:39.790844 (+ 383us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:52:39.790964 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:52:39.791033 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:39.791153 (+ 120us) server_negotiation.cc:299] Negotiation successful
0619 18:52:39.791218 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
W20260619 18:52:39.798873 27078 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:52:39.797200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38414 (local address 127.3.243.124:33211)
0619 18:52:39.797371 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:39.797375 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:39.797671 (+ 296us) server_negotiation.cc:407] Connection header received
0619 18:52:39.797811 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:39.797838 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:39.797911 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:39.797994 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:52:39.798675 (+ 681us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38414: BlockingRecv error: recv got EOF from 127.0.0.1:38414 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 18:52:39.928233 27119 catalog_manager.cc:1259] Loaded cluster ID: e492603c31ab48fbb39497e3f6e66023
I20260619 18:52:39.928328 27119 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 880045b44ae8492580bb907d77e97ea7: loading cluster ID for follower catalog manager: success
I20260619 18:52:39.930981 27119 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 880045b44ae8492580bb907d77e97ea7: acquiring CA information for follower catalog manager: success
I20260619 18:52:39.932783 27119 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 880045b44ae8492580bb907d77e97ea7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 18:52:40.063520 27082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:40.058656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47650 (local address 127.3.243.125:37293)
0619 18:52:40.058898 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:40.058905 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:40.059095 (+ 190us) server_negotiation.cc:407] Connection header received
0619 18:52:40.059258 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:40.059263 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:40.059341 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:40.059421 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:52:40.059845 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:40.060576 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:40.062013 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:40.062426 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:40.062522 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:40.062873 (+ 351us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:52:40.063001 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:52:40.063066 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:40.063170 (+ 104us) server_negotiation.cc:299] Negotiation successful
0619 18:52:40.063233 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260619 18:52:40.074906 27078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:40.069464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38416 (local address 127.3.243.124:33211)
0619 18:52:40.069639 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:40.069645 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:40.069963 (+ 318us) server_negotiation.cc:407] Connection header received
0619 18:52:40.070135 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:40.070141 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:40.070296 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:40.070390 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:52:40.070756 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:40.071430 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:40.072831 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:40.073256 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:40.073365 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:40.073853 (+ 488us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:52:40.074360 (+ 507us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:52:40.074501 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:40.074577 (+ 76us) server_negotiation.cc:299] Negotiation successful
0619 18:52:40.074656 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 18:52:40.086637 27082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:40.079887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47652 (local address 127.3.243.125:37293)
0619 18:52:40.080063 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:40.080068 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:40.080634 (+ 566us) server_negotiation.cc:407] Connection header received
0619 18:52:40.080790 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:40.080796 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:40.080887 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:40.080988 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:52:40.081458 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:40.082571 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:40.084614 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:40.085304 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:40.085429 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:40.085880 (+ 451us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:52:40.086035 (+ 155us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:52:40.086113 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:40.086265 (+ 152us) server_negotiation.cc:299] Negotiation successful
0619 18:52:40.086344 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 18:52:40.092653 27078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:40.080274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38422 (local address 127.3.243.124:33211)
0619 18:52:40.080435 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:40.080450 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:40.086688 (+ 6238us) server_negotiation.cc:407] Connection header received
0619 18:52:40.087098 (+ 410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:40.087105 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:40.087190 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:40.087251 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:52:40.087831 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:40.088967 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:40.090946 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:40.091440 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:40.091555 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:40.091914 (+ 359us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:52:40.092021 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:52:40.092086 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:40.092254 (+ 168us) server_negotiation.cc:299] Negotiation successful
0619 18:52:40.092355 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 18:52:40.096382 27257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:52:40.080000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47134 (local address 127.3.243.126:38079)
0619 18:52:40.090660 (+ 10660us) server_negotiation.cc:206] Beginning negotiation
0619 18:52:40.090666 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:52:40.091269 (+ 603us) server_negotiation.cc:407] Connection header received
0619 18:52:40.091420 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:52:40.091425 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:52:40.091650 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:52:40.091799 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:52:40.092272 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:40.093216 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:40.094592 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:52:40.095018 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:52:40.095114 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:52:40.095500 (+ 386us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:52:40.095869 (+ 369us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:52:40.095960 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 18:52:40.096038 (+ 78us) server_negotiation.cc:299] Negotiation successful
0619 18:52:40.096111 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10531,"thread_start_us":10682,"threads_started":1}
I20260619 18:52:40.100627 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 26860
I20260619 18:52:40.101622 26922 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:52:40.210680 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 26860
I20260619 18:52:40.231812 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 26934
I20260619 18:52:40.232946 26996 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:52:40.342244 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 26934
I20260619 18:52:40.358685 4045 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu pid 27008
I20260619 18:52:40.359751 27070 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 18:52:40.465329 4045 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRgbpDZ/build/asan/bin/kudu with pid 27008
2026-06-19T18:52:40Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4340 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9060 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (349715 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST